2023-12-15T22:40:40.644 INFO:root:teuthology version: 0.0.1.dev216+g5f3731e 2023-12-15T22:40:40.644 DEBUG:teuthology.run:Teuthology command: teuthology --description crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects} --verbose --archive /home/teuthworker/archive/sjust-2023-12-15_22:32:12-crimson-rados-wip-sjust-crimson-testing-2023-12-15-distro-default-smithi/7493355 --owner scheduled_sjust@teuthology --name sjust-2023-12-15_22:32:12-crimson-rados-wip-sjust-crimson-testing-2023-12-15-distro-default-smithi -- /home/teuthworker/archive/sjust-2023-12-15_22:32:12-crimson-rados-wip-sjust-crimson-testing-2023-12-15-distro-default-smithi/7493355/orig.config.yaml 2023-12-15T22:40:40.678 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-15T22:40:40.770 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/sjust-2023-12-15_22:32:12-crimson-rados-wip-sjust-crimson-testing-2023-12-15-distro-default-smithi/7493355 branch: wip-sjust-crimson-testing-2023-12-15 description: crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects} email: null first_in_suite: false job_id: '7493355' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: sjust-2023-12-15_22:32:12-crimson-rados-wip-sjust-crimson-testing-2023-12-15-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-sjust-crimson-testing-2023-12-15 ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 osd pool default crimson: true paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson seastar smp: 3 debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd failsafe full ratio: 0.95 osd max backfills: 3 osd max markdown count: 1000 osd objectstore: bluestore osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from sha1: cd750a7fc894b5dea3d055d595c0da884f63f94f wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: cd750a7fc894b5dea3d055d595c0da884f63f94f thrashosds: min_in: 4 workunit: branch: wip-sjust-crimson-testing-2023-12-15 env: CRIMSON_COMPAT: '1' sha1: cd750a7fc894b5dea3d055d595c0da884f63f94f owner: scheduled_sjust@teuthology priority: 51 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - osd.0 - osd.1 - client.0 - node-exporter.a - - mgr.x - osd.2 - osd.3 - client.1 - prometheus.a - node-exporter.b seed: 6485 sha1: cd750a7fc894b5dea3d055d595c0da884f63f94f sleep_before_teardown: 0 suite: crimson-rados suite_branch: wip-sjust-crimson-testing-2023-12-15 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_cd750a7fc894b5dea3d055d595c0da884f63f94f/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: cd750a7fc894b5dea3d055d595c0da884f63f94f targets: smithi071.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAlS1MwrDgn45o6kUzFJ0kxTcv+jWab7PAGlEmLqOivn4AQC/AmL5RsWCnRViqkqngl33KaeYxO5ZMD18jIkhMY= smithi072.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKMWi4rHWHdGiQVIsrAthgifQCjM9Y5hC8faCromrItnrKy59Pj4KasGyNB+439jYnGaYGZ5pFpQo2xCakbKqVY= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson - thrashosds: ceph_objectstore_tool: false chance_down: 0 chance_inject_pause_short: 0 chance_thrash_cluster_full: 0 chance_thrash_pg_upmap: 0 dump_ops_enable: false min_in: 3 noscrub_toggle_delay: 0 reweight_osd: 0 sighup_delay: 0 thrash_primary_affinity: false timeout: 2400 - rados: clients: - client.0 objects: 50 op_weights: copy_from: 0 delete: 50 read: 100 rollback: 0 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73 timestamp: 2023-12-15_22:32:12 tube: smithi user: sjust verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1485078 2023-12-15T22:40:40.770 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_cd750a7fc894b5dea3d055d595c0da884f63f94f/qa; will attempt to use it 2023-12-15T22:40:40.771 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_cd750a7fc894b5dea3d055d595c0da884f63f94f/qa/tasks 2023-12-15T22:40:40.771 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-15T22:40:40.773 INFO:teuthology.task.internal:Checking packages... 2023-12-15T22:40:40.793 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash 'cd750a7fc894b5dea3d055d595c0da884f63f94f' 2023-12-15T22:40:40.793 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-15T22:40:40.794 INFO:teuthology.packaging:ref: None 2023-12-15T22:40:40.794 INFO:teuthology.packaging:tag: None 2023-12-15T22:40:40.794 INFO:teuthology.packaging:branch: wip-sjust-crimson-testing-2023-12-15 2023-12-15T22:40:40.794 INFO:teuthology.packaging:sha1: cd750a7fc894b5dea3d055d595c0da884f63f94f 2023-12-15T22:40:40.794 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-sjust-crimson-testing-2023-12-15 2023-12-15T22:40:41.010 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-201.gcd750a7f 2023-12-15T22:40:41.011 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-15T22:40:41.020 INFO:teuthology.task.internal:no buildpackages task found 2023-12-15T22:40:41.020 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-15T22:40:41.061 INFO:teuthology.task.internal:Saving configuration 2023-12-15T22:40:41.075 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-15T22:40:41.085 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-15T22:40:41.114 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi071.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sjust-2023-12-15_22:32:12-crimson-rados-wip-sjust-crimson-testing-2023-12-15-distro-default-smithi/7493355', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-15 22:34:16.922658', 'locked_by': 'scheduled_sjust@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAlS1MwrDgn45o6kUzFJ0kxTcv+jWab7PAGlEmLqOivn4AQC/AmL5RsWCnRViqkqngl33KaeYxO5ZMD18jIkhMY='} 2023-12-15T22:40:41.135 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi072.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sjust-2023-12-15_22:32:12-crimson-rados-wip-sjust-crimson-testing-2023-12-15-distro-default-smithi/7493355', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-15 22:34:16.921234', 'locked_by': 'scheduled_sjust@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKMWi4rHWHdGiQVIsrAthgifQCjM9Y5hC8faCromrItnrKy59Pj4KasGyNB+439jYnGaYGZ5pFpQo2xCakbKqVY='} 2023-12-15T22:40:41.135 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-15T22:40:41.143 INFO:teuthology.task.internal:roles: ubuntu@smithi071.front.sepia.ceph.com - ['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'] 2023-12-15T22:40:41.143 INFO:teuthology.task.internal:roles: ubuntu@smithi072.front.sepia.ceph.com - ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-12-15T22:40:41.144 INFO:teuthology.run_tasks:Running task console_log... 2023-12-15T22:40:41.228 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6260f87dc0>, signals=[15]) 2023-12-15T22:40:41.228 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-15T22:40:41.237 INFO:teuthology.task.internal:Opening connections... 2023-12-15T22:40:41.237 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi071.front.sepia.ceph.com 2023-12-15T22:40:41.238 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-15T22:40:41.323 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi072.front.sepia.ceph.com 2023-12-15T22:40:41.324 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-15T22:40:41.406 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-15T22:40:41.412 DEBUG:teuthology.orchestra.run.smithi071:> uname -m 2023-12-15T22:40:41.429 INFO:teuthology.orchestra.run.smithi071.stdout:x86_64 2023-12-15T22:40:41.430 DEBUG:teuthology.orchestra.run.smithi071:> cat /etc/os-release 2023-12-15T22:40:41.486 INFO:teuthology.orchestra.run.smithi071.stdout:NAME="CentOS Stream" 2023-12-15T22:40:41.486 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION="9" 2023-12-15T22:40:41.486 INFO:teuthology.orchestra.run.smithi071.stdout:ID="centos" 2023-12-15T22:40:41.486 INFO:teuthology.orchestra.run.smithi071.stdout:ID_LIKE="rhel fedora" 2023-12-15T22:40:41.486 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION_ID="9" 2023-12-15T22:40:41.486 INFO:teuthology.orchestra.run.smithi071.stdout:PLATFORM_ID="platform:el9" 2023-12-15T22:40:41.486 INFO:teuthology.orchestra.run.smithi071.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-15T22:40:41.486 INFO:teuthology.orchestra.run.smithi071.stdout:ANSI_COLOR="0;31" 2023-12-15T22:40:41.487 INFO:teuthology.orchestra.run.smithi071.stdout:LOGO="fedora-logo-icon" 2023-12-15T22:40:41.487 INFO:teuthology.orchestra.run.smithi071.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-15T22:40:41.487 INFO:teuthology.orchestra.run.smithi071.stdout:HOME_URL="https://centos.org/" 2023-12-15T22:40:41.487 INFO:teuthology.orchestra.run.smithi071.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-15T22:40:41.487 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-15T22:40:41.487 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-15T22:40:41.487 INFO:teuthology.lock.ops:Updating smithi071.front.sepia.ceph.com on lock server 2023-12-15T22:40:41.512 DEBUG:teuthology.orchestra.run.smithi072:> uname -m 2023-12-15T22:40:41.530 INFO:teuthology.orchestra.run.smithi072.stdout:x86_64 2023-12-15T22:40:41.530 DEBUG:teuthology.orchestra.run.smithi072:> cat /etc/os-release 2023-12-15T22:40:41.587 INFO:teuthology.orchestra.run.smithi072.stdout:NAME="CentOS Stream" 2023-12-15T22:40:41.587 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION="9" 2023-12-15T22:40:41.587 INFO:teuthology.orchestra.run.smithi072.stdout:ID="centos" 2023-12-15T22:40:41.587 INFO:teuthology.orchestra.run.smithi072.stdout:ID_LIKE="rhel fedora" 2023-12-15T22:40:41.587 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION_ID="9" 2023-12-15T22:40:41.587 INFO:teuthology.orchestra.run.smithi072.stdout:PLATFORM_ID="platform:el9" 2023-12-15T22:40:41.587 INFO:teuthology.orchestra.run.smithi072.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-15T22:40:41.587 INFO:teuthology.orchestra.run.smithi072.stdout:ANSI_COLOR="0;31" 2023-12-15T22:40:41.587 INFO:teuthology.orchestra.run.smithi072.stdout:LOGO="fedora-logo-icon" 2023-12-15T22:40:41.587 INFO:teuthology.orchestra.run.smithi072.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-15T22:40:41.587 INFO:teuthology.orchestra.run.smithi072.stdout:HOME_URL="https://centos.org/" 2023-12-15T22:40:41.587 INFO:teuthology.orchestra.run.smithi072.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-15T22:40:41.588 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-15T22:40:41.588 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-15T22:40:41.588 INFO:teuthology.lock.ops:Updating smithi072.front.sepia.ceph.com on lock server 2023-12-15T22:40:41.612 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-15T22:40:41.622 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-15T22:40:41.644 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-15T22:40:41.644 DEBUG:teuthology.orchestra.run.smithi071:> test '!' -e /home/ubuntu/cephtest 2023-12-15T22:40:41.647 DEBUG:teuthology.orchestra.run.smithi072:> test '!' -e /home/ubuntu/cephtest 2023-12-15T22:40:41.662 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-15T22:40:41.669 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-15T22:40:41.669 DEBUG:teuthology.orchestra.run.smithi071:> test -z $(ls -A /var/lib/ceph) 2023-12-15T22:40:41.703 DEBUG:teuthology.orchestra.run.smithi072:> test -z $(ls -A /var/lib/ceph) 2023-12-15T22:40:41.790 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-15T22:40:41.842 INFO:teuthology.run_tasks:Running task kernel... 2023-12-15T22:40:41.857 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-15T22:40:41.857 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-15T22:40:41.857 DEBUG:teuthology.orchestra.run.smithi071:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-15T22:40:41.857 DEBUG:teuthology.orchestra.run.smithi072:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-15T22:40:41.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-15T22:40:41.874 DEBUG:teuthology.orchestra.run.smithi071:> uname -r 2023-12-15T22:40:41.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-15T22:40:41.875 DEBUG:teuthology.orchestra.run.smithi072:> uname -r 2023-12-15T22:40:41.930 INFO:teuthology.orchestra.run.smithi071.stdout:5.14.0-361.el9.x86_64 2023-12-15T22:40:41.930 INFO:teuthology.task.kernel:Running kernel on smithi071: 5.14.0-361.el9.x86_64 2023-12-15T22:40:41.930 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum install -y kernel 2023-12-15T22:40:41.931 INFO:teuthology.orchestra.run.smithi072.stdout:5.14.0-361.el9.x86_64 2023-12-15T22:40:41.932 INFO:teuthology.task.kernel:Running kernel on smithi072: 5.14.0-361.el9.x86_64 2023-12-15T22:40:41.932 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum install -y kernel 2023-12-15T22:40:44.046 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-9 - AppStream 15 MB/s | 6.4 MB 00:00 2023-12-15T22:40:44.936 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS-9 - AppStream 16 MB/s | 6.4 MB 00:00 2023-12-15T22:40:45.355 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-9 - BaseOS 8.0 MB/s | 2.0 MB 00:00 2023-12-15T22:40:46.270 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS-9 - BaseOS 8.0 MB/s | 2.0 MB 00:00 2023-12-15T22:40:46.404 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 7.9 MB 00:00 2023-12-15T22:40:47.872 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - BaseOS 6.8 MB/s | 7.9 MB 00:01 2023-12-15T22:40:49.457 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - AppStream 9.1 MB/s | 18 MB 00:01 2023-12-15T22:40:50.517 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - AppStream 11 MB/s | 18 MB 00:01 2023-12-15T22:40:54.751 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - CRB 6.5 MB/s | 5.8 MB 00:00 2023-12-15T22:40:55.439 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - CRB 2.3 MB/s | 5.8 MB 00:02 2023-12-15T22:40:56.038 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - Extras packages 70 kB/s | 15 kB 00:00 2023-12-15T22:40:57.273 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - Extras packages 20 kB/s | 15 kB 00:00 2023-12-15T22:40:59.018 INFO:teuthology.orchestra.run.smithi071.stdout:Extra Packages for Enterprise Linux 12 MB/s | 20 MB 00:01 2023-12-15T22:41:03.582 INFO:teuthology.orchestra.run.smithi071.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2023-12-15T22:41:05.560 INFO:teuthology.orchestra.run.smithi072.stdout:Extra Packages for Enterprise Linux 2.1 MB/s | 20 MB 00:09 2023-12-15T22:41:05.747 INFO:teuthology.orchestra.run.smithi071.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-15T22:41:05.747 INFO:teuthology.orchestra.run.smithi071.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-15T22:41:05.842 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-12-15T22:41:05.849 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-12-15T22:41:05.849 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2023-12-15T22:41:05.849 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-12-15T22:41:05.849 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-12-15T22:41:05.849 INFO:teuthology.orchestra.run.smithi071.stdout: kernel x86_64 5.14.0-391.el9 baseos 5.6 M 2023-12-15T22:41:05.849 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2023-12-15T22:41:05.850 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-core x86_64 5.14.0-391.el9 baseos 20 M 2023-12-15T22:41:05.850 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules x86_64 5.14.0-391.el9 baseos 38 M 2023-12-15T22:41:05.850 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules-core x86_64 5.14.0-391.el9 baseos 32 M 2023-12-15T22:41:05.850 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:41:05.850 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-12-15T22:41:05.850 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-12-15T22:41:05.850 INFO:teuthology.orchestra.run.smithi071.stdout:Install 4 Packages 2023-12-15T22:41:05.850 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:41:05.850 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 96 M 2023-12-15T22:41:05.850 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 121 M 2023-12-15T22:41:05.851 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-12-15T22:41:06.400 INFO:teuthology.orchestra.run.smithi071.stdout:(1/4): kernel-5.14.0-391.el9.x86_64.rpm 21 MB/s | 5.6 MB 00:00 2023-12-15T22:41:06.734 INFO:teuthology.orchestra.run.smithi071.stdout:(2/4): kernel-core-5.14.0-391.el9.x86_64.rpm 34 MB/s | 20 MB 00:00 2023-12-15T22:41:07.251 INFO:teuthology.orchestra.run.smithi071.stdout:(3/4): kernel-modules-5.14.0-391.el9.x86_64.rpm 34 MB/s | 38 MB 00:01 2023-12-15T22:41:07.551 INFO:teuthology.orchestra.run.smithi071.stdout:(4/4): kernel-modules-core-5.14.0-391.el9.x86_6 28 MB/s | 32 MB 00:01 2023-12-15T22:41:07.551 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-12-15T22:41:07.551 INFO:teuthology.orchestra.run.smithi071.stdout:Total 57 MB/s | 96 MB 00:01 2023-12-15T22:41:08.410 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-12-15T22:41:08.771 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-12-15T22:41:08.771 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-12-15T22:41:10.086 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-12-15T22:41:10.086 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-12-15T22:41:10.190 INFO:teuthology.orchestra.run.smithi072.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2023-12-15T22:41:12.365 INFO:teuthology.orchestra.run.smithi072.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-15T22:41:12.365 INFO:teuthology.orchestra.run.smithi072.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-15T22:41:12.462 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-12-15T22:41:12.477 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-12-15T22:41:12.477 INFO:teuthology.orchestra.run.smithi072.stdout: Package Architecture Version Repository Size 2023-12-15T22:41:12.477 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-12-15T22:41:12.477 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2023-12-15T22:41:12.477 INFO:teuthology.orchestra.run.smithi072.stdout: kernel x86_64 5.14.0-391.el9 baseos 5.6 M 2023-12-15T22:41:12.477 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2023-12-15T22:41:12.477 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-core x86_64 5.14.0-391.el9 baseos 20 M 2023-12-15T22:41:12.477 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-modules x86_64 5.14.0-391.el9 baseos 38 M 2023-12-15T22:41:12.478 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-modules-core x86_64 5.14.0-391.el9 baseos 32 M 2023-12-15T22:41:12.478 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:41:12.478 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2023-12-15T22:41:12.478 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-12-15T22:41:12.478 INFO:teuthology.orchestra.run.smithi072.stdout:Install 4 Packages 2023-12-15T22:41:12.478 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:41:12.478 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 96 M 2023-12-15T22:41:12.478 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 121 M 2023-12-15T22:41:12.478 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2023-12-15T22:41:12.928 INFO:teuthology.orchestra.run.smithi072.stdout:(1/4): kernel-5.14.0-391.el9.x86_64.rpm 17 MB/s | 5.6 MB 00:00 2023-12-15T22:41:13.011 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-12-15T22:41:13.321 INFO:teuthology.orchestra.run.smithi072.stdout:(2/4): kernel-core-5.14.0-391.el9.x86_64.rpm 28 MB/s | 20 MB 00:00 2023-12-15T22:41:13.663 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-modules-core-5.14.0-391.el9.x86_64 1/4 2023-12-15T22:41:13.794 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-15T22:41:14.046 INFO:teuthology.orchestra.run.smithi072.stdout:(3/4): kernel-modules-5.14.0-391.el9.x86_64.rpm 26 MB/s | 38 MB 00:01 2023-12-15T22:41:14.555 INFO:teuthology.orchestra.run.smithi072.stdout:(4/4): kernel-modules-core-5.14.0-391.el9.x86_6 20 MB/s | 32 MB 00:01 2023-12-15T22:41:14.555 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2023-12-15T22:41:14.555 INFO:teuthology.orchestra.run.smithi072.stdout:Total 46 MB/s | 96 MB 00:02 2023-12-15T22:41:15.415 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2023-12-15T22:41:15.793 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2023-12-15T22:41:15.793 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2023-12-15T22:41:16.556 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-15T22:41:16.740 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-15T22:41:17.097 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2023-12-15T22:41:17.098 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2023-12-15T22:41:19.734 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2023-12-15T22:41:20.456 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : kernel-modules-core-5.14.0-391.el9.x86_64 1/4 2023-12-15T22:41:20.627 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-15T22:41:21.340 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-15T22:41:21.533 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-5.14.0-391.el9.x86_64 4/4 2023-12-15T22:41:23.389 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-15T22:41:23.654 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-15T22:41:26.161 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-15T22:41:28.276 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-15T22:41:28.659 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : kernel-5.14.0-391.el9.x86_64 4/4 2023-12-15T22:41:33.303 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-15T22:41:54.877 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 4/4 2023-12-15T22:41:54.920 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 4/4 2023-12-15T22:41:55.402 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-5.14.0-391.el9.x86_64 4/4 2023-12-15T22:41:55.402 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-5.14.0-391.el9.x86_64 1/4 2023-12-15T22:41:55.402 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-15T22:41:55.402 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-15T22:41:55.788 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-15T22:41:55.789 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:41:55.789 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-12-15T22:41:55.789 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-5.14.0-391.el9.x86_64 2023-12-15T22:41:55.789 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-core-5.14.0-391.el9.x86_64 2023-12-15T22:41:55.789 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules-5.14.0-391.el9.x86_64 2023-12-15T22:41:55.789 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules-core-5.14.0-391.el9.x86_64 2023-12-15T22:41:55.789 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:41:55.789 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-12-15T22:41:56.338 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q kernel | sort -rV | head -n 1 2023-12-15T22:41:56.477 INFO:teuthology.orchestra.run.smithi071.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-15T22:41:56.478 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-15T22:41:56.478 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-391.el9.x86_64 2023-12-15T22:41:56.478 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-15T22:41:56.478 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-15T22:41:56.478 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-12-15T22:41:56.478 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi071.front.sepia.ceph.com, path=None, version=distro) 2023-12-15T22:41:56.479 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q kernel | sort -rV | head -n 1 2023-12-15T22:41:56.595 INFO:teuthology.orchestra.run.smithi071.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-15T22:41:56.596 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-15T22:41:56.596 DEBUG:teuthology.orchestra.run.smithi071:> sudo rpm -qi grub2-tools 2023-12-15T22:41:56.639 INFO:teuthology.orchestra.run.smithi071.stdout:Name : grub2-tools 2023-12-15T22:41:56.640 INFO:teuthology.orchestra.run.smithi071.stdout:Epoch : 1 2023-12-15T22:41:56.640 INFO:teuthology.orchestra.run.smithi071.stdout:Version : 2.06 2023-12-15T22:41:56.640 INFO:teuthology.orchestra.run.smithi071.stdout:Release : 68.el9 2023-12-15T22:41:56.640 INFO:teuthology.orchestra.run.smithi071.stdout:Architecture: x86_64 2023-12-15T22:41:56.640 INFO:teuthology.orchestra.run.smithi071.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-15T22:41:56.640 INFO:teuthology.orchestra.run.smithi071.stdout:Group : Unspecified 2023-12-15T22:41:56.640 INFO:teuthology.orchestra.run.smithi071.stdout:Size : 8264254 2023-12-15T22:41:56.640 INFO:teuthology.orchestra.run.smithi071.stdout:License : GPLv3+ 2023-12-15T22:41:56.640 INFO:teuthology.orchestra.run.smithi071.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-15T22:41:56.640 INFO:teuthology.orchestra.run.smithi071.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-15T22:41:56.640 INFO:teuthology.orchestra.run.smithi071.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-15T22:41:56.640 INFO:teuthology.orchestra.run.smithi071.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-15T22:41:56.640 INFO:teuthology.orchestra.run.smithi071.stdout:Packager : builder@centos.org 2023-12-15T22:41:56.641 INFO:teuthology.orchestra.run.smithi071.stdout:Vendor : CentOS 2023-12-15T22:41:56.641 INFO:teuthology.orchestra.run.smithi071.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-15T22:41:56.641 INFO:teuthology.orchestra.run.smithi071.stdout:Summary : Support tools for GRUB. 2023-12-15T22:41:56.641 INFO:teuthology.orchestra.run.smithi071.stdout:Description : 2023-12-15T22:41:56.641 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:41:56.641 INFO:teuthology.orchestra.run.smithi071.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-15T22:41:56.641 INFO:teuthology.orchestra.run.smithi071.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-15T22:41:56.641 INFO:teuthology.orchestra.run.smithi071.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-15T22:41:56.641 INFO:teuthology.orchestra.run.smithi071.stdout:hardware devices. 2023-12-15T22:41:56.641 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:41:56.641 INFO:teuthology.orchestra.run.smithi071.stdout:This subpackage provides tools for support of all platforms. 2023-12-15T22:41:56.642 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-15T22:41:56.642 INFO:teuthology.task.kernel:Updating grub on smithi071 to boot 5.14.0-391.el9.x86_64 2023-12-15T22:41:56.642 DEBUG:teuthology.orchestra.run.smithi071:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-15T22:41:57.214 INFO:teuthology.orchestra.run.smithi071.stderr:Generating grub configuration file ... 2023-12-15T22:41:58.561 INFO:teuthology.orchestra.run.smithi071.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-15T22:41:58.613 INFO:teuthology.orchestra.run.smithi071.stderr:done 2023-12-15T22:41:58.615 DEBUG:teuthology.orchestra.run.smithi071:> mktemp 2023-12-15T22:41:58.630 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.jl8lgFpgPO 2023-12-15T22:41:58.630 DEBUG:teuthology.orchestra.run.smithi071:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.jl8lgFpgPO 2023-12-15T22:41:58.704 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.jl8lgFpgPO 2023-12-15T22:41:58.847 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.jl8lgFpgPO is 6KB 2023-12-15T22:41:58.895 DEBUG:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.jl8lgFpgPO 2023-12-15T22:41:58.909 DEBUG:teuthology.orchestra.run.smithi071:> sudo /bin/ls /boot/loader/entries || true 2023-12-15T22:41:58.974 INFO:teuthology.orchestra.run.smithi071.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-15T22:41:58.974 INFO:teuthology.orchestra.run.smithi071.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-15T22:41:58.974 INFO:teuthology.orchestra.run.smithi071.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-15T22:41:58.974 INFO:teuthology.orchestra.run.smithi071.stdout:15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64.conf 2023-12-15T22:41:58.976 DEBUG:teuthology.orchestra.run.smithi071:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64 2023-12-15T22:41:59.134 DEBUG:teuthology.orchestra.run.smithi071:> sudo shutdown -r now 2023-12-15T22:42:04.014 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 4/4 2023-12-15T22:42:04.056 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 4/4 2023-12-15T22:42:04.414 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-5.14.0-391.el9.x86_64 4/4 2023-12-15T22:42:04.414 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : kernel-5.14.0-391.el9.x86_64 1/4 2023-12-15T22:42:04.414 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-15T22:42:04.414 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-15T22:42:04.859 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-15T22:42:04.860 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:42:04.860 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2023-12-15T22:42:04.860 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-5.14.0-391.el9.x86_64 2023-12-15T22:42:04.860 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-core-5.14.0-391.el9.x86_64 2023-12-15T22:42:04.860 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-modules-5.14.0-391.el9.x86_64 2023-12-15T22:42:04.860 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-modules-core-5.14.0-391.el9.x86_64 2023-12-15T22:42:04.860 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:42:04.860 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-12-15T22:42:05.577 DEBUG:teuthology.orchestra.run.smithi072:> rpm -q kernel | sort -rV | head -n 1 2023-12-15T22:42:05.786 INFO:teuthology.orchestra.run.smithi072.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-15T22:42:05.786 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-15T22:42:05.787 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-391.el9.x86_64 2023-12-15T22:42:05.787 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-15T22:42:05.787 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-15T22:42:05.787 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2023-12-15T22:42:05.787 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi072.front.sepia.ceph.com, path=None, version=distro) 2023-12-15T22:42:05.787 DEBUG:teuthology.orchestra.run.smithi072:> rpm -q kernel | sort -rV | head -n 1 2023-12-15T22:42:05.905 INFO:teuthology.orchestra.run.smithi072.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-15T22:42:05.905 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-15T22:42:05.905 DEBUG:teuthology.orchestra.run.smithi072:> sudo rpm -qi grub2-tools 2023-12-15T22:42:05.948 INFO:teuthology.orchestra.run.smithi072.stdout:Name : grub2-tools 2023-12-15T22:42:05.948 INFO:teuthology.orchestra.run.smithi072.stdout:Epoch : 1 2023-12-15T22:42:05.948 INFO:teuthology.orchestra.run.smithi072.stdout:Version : 2.06 2023-12-15T22:42:05.948 INFO:teuthology.orchestra.run.smithi072.stdout:Release : 68.el9 2023-12-15T22:42:05.948 INFO:teuthology.orchestra.run.smithi072.stdout:Architecture: x86_64 2023-12-15T22:42:05.948 INFO:teuthology.orchestra.run.smithi072.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-15T22:42:05.948 INFO:teuthology.orchestra.run.smithi072.stdout:Group : Unspecified 2023-12-15T22:42:05.948 INFO:teuthology.orchestra.run.smithi072.stdout:Size : 8264254 2023-12-15T22:42:05.948 INFO:teuthology.orchestra.run.smithi072.stdout:License : GPLv3+ 2023-12-15T22:42:05.948 INFO:teuthology.orchestra.run.smithi072.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-15T22:42:05.948 INFO:teuthology.orchestra.run.smithi072.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-15T22:42:05.949 INFO:teuthology.orchestra.run.smithi072.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-15T22:42:05.949 INFO:teuthology.orchestra.run.smithi072.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-15T22:42:05.949 INFO:teuthology.orchestra.run.smithi072.stdout:Packager : builder@centos.org 2023-12-15T22:42:05.949 INFO:teuthology.orchestra.run.smithi072.stdout:Vendor : CentOS 2023-12-15T22:42:05.949 INFO:teuthology.orchestra.run.smithi072.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-15T22:42:05.949 INFO:teuthology.orchestra.run.smithi072.stdout:Summary : Support tools for GRUB. 2023-12-15T22:42:05.949 INFO:teuthology.orchestra.run.smithi072.stdout:Description : 2023-12-15T22:42:05.949 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:42:05.949 INFO:teuthology.orchestra.run.smithi072.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-15T22:42:05.949 INFO:teuthology.orchestra.run.smithi072.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-15T22:42:05.949 INFO:teuthology.orchestra.run.smithi072.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-15T22:42:05.949 INFO:teuthology.orchestra.run.smithi072.stdout:hardware devices. 2023-12-15T22:42:05.949 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:42:05.949 INFO:teuthology.orchestra.run.smithi072.stdout:This subpackage provides tools for support of all platforms. 2023-12-15T22:42:05.950 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-15T22:42:05.950 INFO:teuthology.task.kernel:Updating grub on smithi072 to boot 5.14.0-391.el9.x86_64 2023-12-15T22:42:05.950 DEBUG:teuthology.orchestra.run.smithi072:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-15T22:42:06.626 INFO:teuthology.orchestra.run.smithi072.stderr:Generating grub configuration file ... 2023-12-15T22:42:08.136 INFO:teuthology.orchestra.run.smithi072.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-15T22:42:08.162 INFO:teuthology.orchestra.run.smithi072.stderr:done 2023-12-15T22:42:08.165 DEBUG:teuthology.orchestra.run.smithi072:> mktemp 2023-12-15T22:42:08.190 INFO:teuthology.orchestra.run.smithi072.stdout:/tmp/tmp.wFzZqZobro 2023-12-15T22:42:08.191 DEBUG:teuthology.orchestra.run.smithi072:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.wFzZqZobro 2023-12-15T22:42:08.259 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 0666 /tmp/tmp.wFzZqZobro 2023-12-15T22:42:08.401 DEBUG:teuthology.orchestra.remote:smithi072:/tmp/tmp.wFzZqZobro is 6KB 2023-12-15T22:42:08.449 DEBUG:teuthology.orchestra.run.smithi072:> rm -fr /tmp/tmp.wFzZqZobro 2023-12-15T22:42:08.463 DEBUG:teuthology.orchestra.run.smithi072:> sudo /bin/ls /boot/loader/entries || true 2023-12-15T22:42:08.529 INFO:teuthology.orchestra.run.smithi072.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-15T22:42:08.529 INFO:teuthology.orchestra.run.smithi072.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-15T22:42:08.529 INFO:teuthology.orchestra.run.smithi072.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-15T22:42:08.529 INFO:teuthology.orchestra.run.smithi072.stdout:15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64.conf 2023-12-15T22:42:08.530 DEBUG:teuthology.orchestra.run.smithi072:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64 2023-12-15T22:42:08.681 DEBUG:teuthology.orchestra.run.smithi072:> sudo shutdown -r now 2023-12-15T22:42:29.152 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-12-15T22:42:29.153 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2023-12-15T22:42:29.154 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-15T22:42:38.690 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2023-12-15T22:42:38.690 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2023-12-15T22:42:38.690 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-15T22:43:03.830 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2023-12-15T22:43:12.853 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2023-12-15T22:43:12.854 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-15T22:43:31.217 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2023-12-15T22:43:38.698 DEBUG:teuthology.orchestra.remote:timed out 2023-12-15T22:43:43.221 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2023-12-15T22:43:43.222 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-15T22:43:43.645 DEBUG:teuthology.orchestra.run.smithi071:> true 2023-12-15T22:43:44.033 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi071.front.sepia.ceph.com' 2023-12-15T22:43:44.033 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-391.el9.x86_64"... 2023-12-15T22:43:44.033 DEBUG:teuthology.orchestra.run.smithi071:> uname -r 2023-12-15T22:43:44.047 INFO:teuthology.orchestra.run.smithi071.stdout:5.14.0-391.el9.x86_64 2023-12-15T22:43:44.047 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-391.el9.x86_64 vs 5.14.0-391.el9.x86_64 2023-12-15T22:43:44.047 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-15T22:43:44.047 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-15T22:43:45.048 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-12-15T22:43:45.048 DEBUG:teuthology.orchestra.run.smithi071:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-15T22:43:45.109 INFO:teuthology.orchestra.run.smithi071.stdout:ttyS1 2023-12-15T22:43:45.145 DEBUG:teuthology.parallel:result is None 2023-12-15T22:43:47.699 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2023-12-15T22:43:47.700 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-15T22:43:55.278 DEBUG:teuthology.orchestra.run.smithi072:> true 2023-12-15T22:43:55.786 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi072.front.sepia.ceph.com' 2023-12-15T22:43:55.786 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.14.0-391.el9.x86_64"... 2023-12-15T22:43:55.786 DEBUG:teuthology.orchestra.run.smithi072:> uname -r 2023-12-15T22:43:55.800 INFO:teuthology.orchestra.run.smithi072.stdout:5.14.0-391.el9.x86_64 2023-12-15T22:43:55.800 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-391.el9.x86_64 vs 5.14.0-391.el9.x86_64 2023-12-15T22:43:55.801 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-15T22:43:55.801 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-15T22:43:56.802 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2023-12-15T22:43:56.802 DEBUG:teuthology.orchestra.run.smithi072:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-15T22:43:56.907 INFO:teuthology.orchestra.run.smithi072.stdout:ttyS1 2023-12-15T22:43:56.963 DEBUG:teuthology.parallel:result is None 2023-12-15T22:43:56.963 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-15T22:43:56.970 INFO:teuthology.task.internal:Creating test directory... 2023-12-15T22:43:56.971 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-15T22:43:56.974 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-15T22:43:56.989 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-15T22:43:56.996 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-15T22:43:57.039 INFO:teuthology.task.internal:Creating archive directory... 2023-12-15T22:43:57.039 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-15T22:43:57.041 DEBUG:teuthology.orchestra.run.smithi072:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-15T22:43:57.162 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-15T22:43:57.168 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-15T22:43:57.169 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-15T22:43:57.171 DEBUG:teuthology.orchestra.run.smithi072:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-15T22:43:57.202 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-15T22:43:57.212 INFO:teuthology.orchestra.run.smithi072.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-15T22:43:57.216 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-15T22:43:57.226 INFO:teuthology.orchestra.run.smithi072.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-15T22:43:57.227 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-15T22:43:57.234 INFO:teuthology.task.internal:Configuring sudo... 2023-12-15T22:43:57.234 DEBUG:teuthology.orchestra.run.smithi071:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-15T22:43:57.260 DEBUG:teuthology.orchestra.run.smithi072:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-15T22:43:57.298 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-15T22:43:57.307 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-15T22:43:57.307 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-15T22:43:57.330 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-15T22:43:57.356 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-15T22:43:57.428 DEBUG:teuthology.orchestra.run.smithi071:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-15T22:43:57.481 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-15T22:43:57.560 DEBUG:teuthology.orchestra.run.smithi071:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-15T22:43:57.608 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-12-15T22:43:57.608 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-15T22:43:57.675 DEBUG:teuthology.orchestra.run.smithi072:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-15T22:43:57.713 DEBUG:teuthology.orchestra.run.smithi072:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-15T22:43:57.773 DEBUG:teuthology.orchestra.run.smithi072:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-15T22:43:57.855 DEBUG:teuthology.orchestra.run.smithi072:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-15T22:43:57.904 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2023-12-15T22:43:57.904 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-15T22:43:57.977 DEBUG:teuthology.orchestra.run.smithi071:> sudo service rsyslog restart 2023-12-15T22:43:57.979 DEBUG:teuthology.orchestra.run.smithi072:> sudo service rsyslog restart 2023-12-15T22:43:58.022 INFO:teuthology.orchestra.run.smithi071.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-15T22:43:58.126 INFO:teuthology.orchestra.run.smithi072.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-15T22:43:58.530 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-15T22:43:58.538 INFO:teuthology.task.internal:Starting timer... 2023-12-15T22:43:58.538 INFO:teuthology.run_tasks:Running task pcp... 2023-12-15T22:43:58.567 INFO:teuthology.run_tasks:Running task selinux... 2023-12-15T22:43:58.599 DEBUG:teuthology.orchestra.run.smithi071:> sudo service auditd rotate 2023-12-15T22:43:58.656 INFO:teuthology.orchestra.run.smithi071.stdout:Rotating logs: 2023-12-15T22:43:58.658 DEBUG:teuthology.orchestra.run.smithi072:> sudo service auditd rotate 2023-12-15T22:43:58.730 INFO:teuthology.orchestra.run.smithi072.stdout:Rotating logs: 2023-12-15T22:43:58.730 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-15T22:43:58.730 DEBUG:teuthology.orchestra.run.smithi071:> /usr/sbin/getenforce 2023-12-15T22:43:58.751 INFO:teuthology.orchestra.run.smithi071.stdout:Permissive 2023-12-15T22:43:58.751 DEBUG:teuthology.orchestra.run.smithi072:> /usr/sbin/getenforce 2023-12-15T22:43:58.780 INFO:teuthology.orchestra.run.smithi072.stdout:Permissive 2023-12-15T22:43:58.781 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi071.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi072.front.sepia.ceph.com': 'permissive'} 2023-12-15T22:43:58.781 DEBUG:teuthology.orchestra.run.smithi071:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-15T22:43:58.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-15T22:43:58.821 DEBUG:teuthology.orchestra.run.smithi072:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-15T22:43:58.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-15T22:43:58.852 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-15T22:43:58.852 DEBUG:teuthology.orchestra.run.smithi071:> sudo /usr/sbin/setenforce permissive 2023-12-15T22:43:58.894 DEBUG:teuthology.orchestra.run.smithi072:> sudo /usr/sbin/setenforce permissive 2023-12-15T22:43:58.933 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-15T22:43:58.943 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-12-15T22:43:58.943 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-15T22:43:58.965 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-15T22:43:58.969 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi071.front.sepia.ceph.com,smithi072.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-15T22:50:02.106 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi071.front.sepia.ceph.com'), Remote(name='ubuntu@smithi072.front.sepia.ceph.com')] 2023-12-15T22:50:02.107 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2023-12-15T22:50:02.108 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-15T22:50:02.195 DEBUG:teuthology.orchestra.run.smithi071:> true 2023-12-15T22:50:02.259 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi071.front.sepia.ceph.com' 2023-12-15T22:50:02.260 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2023-12-15T22:50:02.260 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-15T22:50:02.343 DEBUG:teuthology.orchestra.run.smithi072:> true 2023-12-15T22:50:02.414 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi072.front.sepia.ceph.com' 2023-12-15T22:50:02.414 INFO:teuthology.run_tasks:Running task clock... 2023-12-15T22:50:02.425 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-15T22:50:02.425 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-15T22:50:02.426 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-15T22:50:02.428 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-15T22:50:02.428 DEBUG:teuthology.orchestra.run.smithi072:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-15T22:50:02.457 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-15T22:50:02.473 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-15T22:50:02.488 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-15T22:50:02.501 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: ntpd: command not found 2023-12-15T22:50:02.505 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-15T22:50:02.515 INFO:teuthology.orchestra.run.smithi071.stdout:506 Cannot talk to daemon 2023-12-15T22:50:02.531 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-15T22:50:02.535 INFO:teuthology.orchestra.run.smithi072.stderr:sudo: ntpd: command not found 2023-12-15T22:50:02.547 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-15T22:50:02.549 INFO:teuthology.orchestra.run.smithi072.stdout:506 Cannot talk to daemon 2023-12-15T22:50:02.565 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-15T22:50:02.582 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-15T22:50:02.590 INFO:teuthology.orchestra.run.smithi071.stderr:bash: line 1: ntpq: command not found 2023-12-15T22:50:02.593 INFO:teuthology.orchestra.run.smithi071.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-15T22:50:02.594 INFO:teuthology.orchestra.run.smithi071.stdout:=============================================================================== 2023-12-15T22:50:02.594 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-15T22:50:02.594 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-15T22:50:02.594 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-15T22:50:02.594 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-15T22:50:02.625 INFO:teuthology.orchestra.run.smithi072.stderr:bash: line 1: ntpq: command not found 2023-12-15T22:50:02.628 INFO:teuthology.orchestra.run.smithi072.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-15T22:50:02.628 INFO:teuthology.orchestra.run.smithi072.stdout:=============================================================================== 2023-12-15T22:50:02.628 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-15T22:50:02.628 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-15T22:50:02.628 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-15T22:50:02.628 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-15T22:50:02.629 INFO:teuthology.run_tasks:Running task install... 2023-12-15T22:50:02.637 DEBUG:teuthology.task.install:project ceph 2023-12-15T22:50:02.637 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': 'cd750a7fc894b5dea3d055d595c0da884f63f94f'}} 2023-12-15T22:50:02.637 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': 'cd750a7fc894b5dea3d055d595c0da884f63f94f'} 2023-12-15T22:50:02.638 INFO:teuthology.task.install:Using flavor: crimson 2023-12-15T22:50:02.643 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-15T22:50:02.643 INFO:teuthology.task.install:extra packages: [] 2023-12-15T22:50:02.644 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'cd750a7fc894b5dea3d055d595c0da884f63f94f', 'tag': None, 'wait_for_package': False} 2023-12-15T22:50:02.644 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=cd750a7fc894b5dea3d055d595c0da884f63f94f 2023-12-15T22:50:02.646 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'cd750a7fc894b5dea3d055d595c0da884f63f94f', 'tag': None, 'wait_for_package': False} 2023-12-15T22:50:02.646 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=cd750a7fc894b5dea3d055d595c0da884f63f94f 2023-12-15T22:50:02.790 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-sjust-crimson-testing-2023-12-15/cd750a7fc894b5dea3d055d595c0da884f63f94f/centos/9/flavors/crimson/ 2023-12-15T22:50:02.790 INFO:teuthology.task.install.rpm:Package version is 19.0.0-201.gcd750a7f 2023-12-15T22:50:02.860 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-sjust-crimson-testing-2023-12-15/cd750a7fc894b5dea3d055d595c0da884f63f94f/centos/9/flavors/crimson/ 2023-12-15T22:50:02.860 INFO:teuthology.task.install.rpm:Package version is 19.0.0-201.gcd750a7f 2023-12-15T22:50:03.040 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-sjust-crimson-testing-2023-12-15/cd750a7fc894b5dea3d055d595c0da884f63f94f/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-sjust-crimson-testing-2023-12-15/cd750a7fc894b5dea3d055d595c0da884f63f94f/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-sjust-crimson-testing-2023-12-15/cd750a7fc894b5dea3d055d595c0da884f63f94f/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-15T22:50:03.040 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2023-12-15T22:50:03.040 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-15T22:50:03.069 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-12-15T22:50:03.069 DEBUG:teuthology.orchestra.run.smithi072:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/cd750a7fc894b5dea3d055d595c0da884f63f94f/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-15T22:50:03.105 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-sjust-crimson-testing-2023-12-15/cd750a7fc894b5dea3d055d595c0da884f63f94f/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-sjust-crimson-testing-2023-12-15/cd750a7fc894b5dea3d055d595c0da884f63f94f/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-sjust-crimson-testing-2023-12-15/cd750a7fc894b5dea3d055d595c0da884f63f94f/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-15T22:50:03.105 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-12-15T22:50:03.105 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-15T22:50:03.135 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-12-15T22:50:03.136 DEBUG:teuthology.orchestra.run.smithi071:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/cd750a7fc894b5dea3d055d595c0da884f63f94f/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-15T22:50:03.137 DEBUG:teuthology.orchestra.run.smithi072:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-15T22:50:03.206 DEBUG:teuthology.orchestra.run.smithi071:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-15T22:50:03.216 DEBUG:teuthology.orchestra.run.smithi072:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-15T22:50:03.281 INFO:teuthology.orchestra.run.smithi072.stdout:check_obsoletes = 1 2023-12-15T22:50:03.283 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum clean all 2023-12-15T22:50:03.287 DEBUG:teuthology.orchestra.run.smithi071:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-15T22:50:03.355 INFO:teuthology.orchestra.run.smithi071.stdout:check_obsoletes = 1 2023-12-15T22:50:03.357 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum clean all 2023-12-15T22:50:03.523 INFO:teuthology.orchestra.run.smithi072.stdout:73 files removed 2023-12-15T22:50:03.549 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-radosgw 2023-12-15T22:50:03.608 INFO:teuthology.orchestra.run.smithi071.stdout:73 files removed 2023-12-15T22:50:03.635 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-radosgw 2023-12-15T22:50:04.517 INFO:teuthology.orchestra.run.smithi072.stdout:ceph packages for x86_64 125 kB/s | 82 kB 00:00 2023-12-15T22:50:04.625 INFO:teuthology.orchestra.run.smithi071.stdout:ceph packages for x86_64 121 kB/s | 82 kB 00:00 2023-12-15T22:50:04.920 INFO:teuthology.orchestra.run.smithi072.stdout:ceph noarch packages 27 kB/s | 9.5 kB 00:00 2023-12-15T22:50:05.142 INFO:teuthology.orchestra.run.smithi071.stdout:ceph noarch packages 20 kB/s | 9.5 kB 00:00 2023-12-15T22:50:05.362 INFO:teuthology.orchestra.run.smithi072.stdout:ceph source packages 4.8 kB/s | 1.9 kB 00:00 2023-12-15T22:50:05.618 INFO:teuthology.orchestra.run.smithi071.stdout:ceph source packages 4.5 kB/s | 1.9 kB 00:00 2023-12-15T22:50:05.891 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS-9 - AppStream 13 MB/s | 6.4 MB 00:00 2023-12-15T22:50:06.007 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-9 - AppStream 18 MB/s | 6.4 MB 00:00 2023-12-15T22:50:07.200 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS-9 - BaseOS 7.6 MB/s | 2.0 MB 00:00 2023-12-15T22:50:07.449 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-9 - BaseOS 5.3 MB/s | 2.0 MB 00:00 2023-12-15T22:50:08.356 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 7.9 MB 00:00 2023-12-15T22:50:08.452 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - BaseOS 8.6 MB/s | 7.9 MB 00:00 2023-12-15T22:50:11.682 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - AppStream 8.0 MB/s | 18 MB 00:02 2023-12-15T22:50:12.038 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - AppStream 7.1 MB/s | 18 MB 00:02 2023-12-15T22:50:15.699 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - CRB 9.5 MB/s | 5.8 MB 00:00 2023-12-15T22:50:17.018 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 5.8 MB 00:01 2023-12-15T22:50:17.500 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 15 kB 00:00 2023-12-15T22:50:18.609 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 15 kB 00:00 2023-12-15T22:50:19.488 INFO:teuthology.orchestra.run.smithi071.stdout:Extra Packages for Enterprise Linux 11 MB/s | 20 MB 00:01 2023-12-15T22:50:19.569 INFO:teuthology.orchestra.run.smithi072.stdout:Extra Packages for Enterprise Linux 23 MB/s | 20 MB 00:00 2023-12-15T22:50:24.148 INFO:teuthology.orchestra.run.smithi072.stdout:lab-extras 40 kB/s | 1.7 kB 00:00 2023-12-15T22:50:24.176 INFO:teuthology.orchestra.run.smithi071.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2023-12-15T22:50:26.663 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-12-15T22:50:26.685 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-12-15T22:50:26.685 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2023-12-15T22:50:26.685 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-12-15T22:50:26.685 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2023-12-15T22:50:26.685 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-radosgw x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 8.3 M 2023-12-15T22:50:26.685 INFO:teuthology.orchestra.run.smithi072.stdout:Upgrading: 2023-12-15T22:50:26.685 INFO:teuthology.orchestra.run.smithi072.stdout: librados2 x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 3.1 M 2023-12-15T22:50:26.685 INFO:teuthology.orchestra.run.smithi072.stdout: librbd1 x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 3.0 M 2023-12-15T22:50:26.685 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2023-12-15T22:50:26.685 INFO:teuthology.orchestra.run.smithi072.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-12-15T22:50:26.686 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-base x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 6.6 M 2023-12-15T22:50:26.686 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-common x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 17 M 2023-12-15T22:50:26.686 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-selinux x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 24 k 2023-12-15T22:50:26.686 INFO:teuthology.orchestra.run.smithi072.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-15T22:50:26.686 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2023-12-15T22:50:26.686 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2023-12-15T22:50:26.686 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs2 x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 728 k 2023-12-15T22:50:26.686 INFO:teuthology.orchestra.run.smithi072.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-15T22:50:26.686 INFO:teuthology.orchestra.run.smithi072.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-15T22:50:26.686 INFO:teuthology.orchestra.run.smithi072.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-15T22:50:26.686 INFO:teuthology.orchestra.run.smithi072.stdout: libradosstriper1 x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 470 k 2023-12-15T22:50:26.686 INFO:teuthology.orchestra.run.smithi072.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-15T22:50:26.686 INFO:teuthology.orchestra.run.smithi072.stdout: librgw2 x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 4.5 M 2023-12-15T22:50:26.686 INFO:teuthology.orchestra.run.smithi072.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-15T22:50:26.686 INFO:teuthology.orchestra.run.smithi072.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-15T22:50:26.686 INFO:teuthology.orchestra.run.smithi072.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2023-12-15T22:50:26.687 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-argparse x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 45 k 2023-12-15T22:50:26.687 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-common x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 130 k 2023-12-15T22:50:26.687 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cephfs x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 161 k 2023-12-15T22:50:26.687 INFO:teuthology.orchestra.run.smithi072.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-15T22:50:26.687 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rados x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 321 k 2023-12-15T22:50:26.687 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rbd x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 298 k 2023-12-15T22:50:26.687 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rgw x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 99 k 2023-12-15T22:50:26.687 INFO:teuthology.orchestra.run.smithi072.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-12-15T22:50:26.687 INFO:teuthology.orchestra.run.smithi072.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-15T22:50:26.687 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:50:26.687 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2023-12-15T22:50:26.687 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-12-15T22:50:26.687 INFO:teuthology.orchestra.run.smithi072.stdout:Install 27 Packages 2023-12-15T22:50:26.687 INFO:teuthology.orchestra.run.smithi072.stdout:Upgrade 2 Packages 2023-12-15T22:50:26.687 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:50:26.688 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 54 M 2023-12-15T22:50:26.688 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2023-12-15T22:50:26.701 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-12-15T22:50:26.705 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-12-15T22:50:26.705 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-12-15T22:50:26.705 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-12-15T22:50:26.705 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-12-15T22:50:26.705 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-radosgw x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 8.3 M 2023-12-15T22:50:26.706 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrading: 2023-12-15T22:50:26.706 INFO:teuthology.orchestra.run.smithi071.stdout: librados2 x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 3.1 M 2023-12-15T22:50:26.706 INFO:teuthology.orchestra.run.smithi071.stdout: librbd1 x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 3.0 M 2023-12-15T22:50:26.706 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2023-12-15T22:50:26.706 INFO:teuthology.orchestra.run.smithi071.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-12-15T22:50:26.706 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-base x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 6.6 M 2023-12-15T22:50:26.706 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-common x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 17 M 2023-12-15T22:50:26.706 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-selinux x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 24 k 2023-12-15T22:50:26.706 INFO:teuthology.orchestra.run.smithi071.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-15T22:50:26.706 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2023-12-15T22:50:26.706 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2023-12-15T22:50:26.706 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs2 x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 728 k 2023-12-15T22:50:26.706 INFO:teuthology.orchestra.run.smithi071.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-15T22:50:26.706 INFO:teuthology.orchestra.run.smithi071.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-15T22:50:26.707 INFO:teuthology.orchestra.run.smithi071.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-15T22:50:26.707 INFO:teuthology.orchestra.run.smithi071.stdout: libradosstriper1 x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 470 k 2023-12-15T22:50:26.707 INFO:teuthology.orchestra.run.smithi071.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-15T22:50:26.707 INFO:teuthology.orchestra.run.smithi071.stdout: librgw2 x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 4.5 M 2023-12-15T22:50:26.707 INFO:teuthology.orchestra.run.smithi071.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-15T22:50:26.707 INFO:teuthology.orchestra.run.smithi071.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-15T22:50:26.707 INFO:teuthology.orchestra.run.smithi071.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2023-12-15T22:50:26.707 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-argparse x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 45 k 2023-12-15T22:50:26.707 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-common x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 130 k 2023-12-15T22:50:26.707 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cephfs x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 161 k 2023-12-15T22:50:26.707 INFO:teuthology.orchestra.run.smithi071.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-15T22:50:26.707 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rados x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 321 k 2023-12-15T22:50:26.707 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rbd x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 298 k 2023-12-15T22:50:26.707 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rgw x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 99 k 2023-12-15T22:50:26.707 INFO:teuthology.orchestra.run.smithi071.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-12-15T22:50:26.708 INFO:teuthology.orchestra.run.smithi071.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-15T22:50:26.708 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:50:26.708 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-12-15T22:50:26.708 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-12-15T22:50:26.708 INFO:teuthology.orchestra.run.smithi071.stdout:Install 27 Packages 2023-12-15T22:50:26.708 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrade 2 Packages 2023-12-15T22:50:26.708 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:50:26.708 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 54 M 2023-12-15T22:50:26.708 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-12-15T22:50:34.767 INFO:teuthology.orchestra.run.smithi072.stdout:(1/29): ceph-base-19.0.0-201.gcd750a7f.el9.x86_ 844 kB/s | 6.6 MB 00:08 2023-12-15T22:50:34.801 INFO:teuthology.orchestra.run.smithi071.stdout:(1/29): ceph-base-19.0.0-201.gcd750a7f.el9.x86_ 850 kB/s | 6.6 MB 00:07 2023-12-15T22:50:34.875 INFO:teuthology.orchestra.run.smithi072.stdout:(2/29): ceph-selinux-19.0.0-201.gcd750a7f.el9.x 222 kB/s | 24 kB 00:00 2023-12-15T22:50:34.909 INFO:teuthology.orchestra.run.smithi071.stdout:(2/29): ceph-selinux-19.0.0-201.gcd750a7f.el9.x 222 kB/s | 24 kB 00:00 2023-12-15T22:50:35.827 INFO:teuthology.orchestra.run.smithi072.stdout:(3/29): libcephfs2-19.0.0-201.gcd750a7f.el9.x86 765 kB/s | 728 kB 00:00 2023-12-15T22:50:36.251 INFO:teuthology.orchestra.run.smithi071.stdout:(3/29): libcephfs2-19.0.0-201.gcd750a7f.el9.x86 543 kB/s | 728 kB 00:01 2023-12-15T22:50:36.351 INFO:teuthology.orchestra.run.smithi071.stdout:(4/29): ceph-radosgw-19.0.0-201.gcd750a7f.el9.x 900 kB/s | 8.3 MB 00:09 2023-12-15T22:50:36.369 INFO:teuthology.orchestra.run.smithi072.stdout:(4/29): libradosstriper1-19.0.0-201.gcd750a7f.e 868 kB/s | 470 kB 00:00 2023-12-15T22:50:37.043 INFO:teuthology.orchestra.run.smithi071.stdout:(5/29): libradosstriper1-19.0.0-201.gcd750a7f.e 594 kB/s | 470 kB 00:00 2023-12-15T22:50:37.153 INFO:teuthology.orchestra.run.smithi071.stdout:(6/29): python3-ceph-argparse-19.0.0-201.gcd750 416 kB/s | 45 kB 00:00 2023-12-15T22:50:37.869 INFO:teuthology.orchestra.run.smithi071.stdout:(7/29): python3-ceph-common-19.0.0-201.gcd750a7 181 kB/s | 130 kB 00:00 2023-12-15T22:50:38.185 INFO:teuthology.orchestra.run.smithi071.stdout:(8/29): python3-cephfs-19.0.0-201.gcd750a7f.el9 510 kB/s | 161 kB 00:00 2023-12-15T22:50:38.752 INFO:teuthology.orchestra.run.smithi071.stdout:(9/29): python3-rados-19.0.0-201.gcd750a7f.el9. 566 kB/s | 321 kB 00:00 2023-12-15T22:50:39.436 INFO:teuthology.orchestra.run.smithi071.stdout:(10/29): python3-rbd-19.0.0-201.gcd750a7f.el9.x 436 kB/s | 298 kB 00:00 2023-12-15T22:50:39.678 INFO:teuthology.orchestra.run.smithi071.stdout:(11/29): python3-rgw-19.0.0-201.gcd750a7f.el9.x 411 kB/s | 99 kB 00:00 2023-12-15T22:50:39.795 INFO:teuthology.orchestra.run.smithi071.stdout:(12/29): boost-program-options-1.75.0-8.el9.x86 906 kB/s | 106 kB 00:00 2023-12-15T22:50:39.837 INFO:teuthology.orchestra.run.smithi071.stdout:(13/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2023-12-15T22:50:39.871 INFO:teuthology.orchestra.run.smithi071.stdout:(14/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2023-12-15T22:50:39.930 INFO:teuthology.orchestra.run.smithi071.stdout:(15/29): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2023-12-15T22:50:39.963 INFO:teuthology.orchestra.run.smithi071.stdout:(16/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 8.3 MB/s | 282 kB 00:00 2023-12-15T22:50:39.997 INFO:teuthology.orchestra.run.smithi071.stdout:(17/29): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2023-12-15T22:50:40.147 INFO:teuthology.orchestra.run.smithi071.stdout:(18/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.0 MB/s | 309 kB 00:00 2023-12-15T22:50:40.323 INFO:teuthology.orchestra.run.smithi071.stdout:(19/29): libarrow-9.0.0-10.el9.x86_64.rpm 25 MB/s | 4.4 MB 00:00 2023-12-15T22:50:40.364 INFO:teuthology.orchestra.run.smithi071.stdout:(20/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 627 kB/s | 26 kB 00:00 2023-12-15T22:50:40.406 INFO:teuthology.orchestra.run.smithi071.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2023-12-15T22:50:40.448 INFO:teuthology.orchestra.run.smithi071.stdout:(22/29): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2023-12-15T22:50:40.515 INFO:teuthology.orchestra.run.smithi071.stdout:(23/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 12 MB/s | 838 kB 00:00 2023-12-15T22:50:40.557 INFO:teuthology.orchestra.run.smithi071.stdout:(24/29): re2-20211101-3.el9.x86_64.rpm 4.5 MB/s | 192 kB 00:00 2023-12-15T22:50:40.624 INFO:teuthology.orchestra.run.smithi071.stdout:(25/29): thrift-0.15.0-2.el9.x86_64.rpm 24 MB/s | 1.6 MB 00:00 2023-12-15T22:50:41.653 INFO:teuthology.orchestra.run.smithi072.stdout:(5/29): librgw2-19.0.0-201.gcd750a7f.el9.x86_64 868 kB/s | 4.5 MB 00:05 2023-12-15T22:50:41.770 INFO:teuthology.orchestra.run.smithi072.stdout:(6/29): python3-ceph-argparse-19.0.0-201.gcd750 387 kB/s | 45 kB 00:00 2023-12-15T22:50:41.979 INFO:teuthology.orchestra.run.smithi072.stdout:(7/29): python3-ceph-common-19.0.0-201.gcd750a7 622 kB/s | 130 kB 00:00 2023-12-15T22:50:42.305 INFO:teuthology.orchestra.run.smithi072.stdout:(8/29): python3-cephfs-19.0.0-201.gcd750a7f.el9 497 kB/s | 161 kB 00:00 2023-12-15T22:50:42.605 INFO:teuthology.orchestra.run.smithi072.stdout:(9/29): ceph-radosgw-19.0.0-201.gcd750a7f.el9.x 539 kB/s | 8.3 MB 00:15 2023-12-15T22:50:42.831 INFO:teuthology.orchestra.run.smithi072.stdout:(10/29): python3-rados-19.0.0-201.gcd750a7f.el9 610 kB/s | 321 kB 00:00 2023-12-15T22:50:43.040 INFO:teuthology.orchestra.run.smithi072.stdout:(11/29): python3-rgw-19.0.0-201.gcd750a7f.el9.x 476 kB/s | 99 kB 00:00 2023-12-15T22:50:43.098 INFO:teuthology.orchestra.run.smithi072.stdout:(12/29): python3-rbd-19.0.0-201.gcd750a7f.el9.x 605 kB/s | 298 kB 00:00 2023-12-15T22:50:43.157 INFO:teuthology.orchestra.run.smithi072.stdout:(13/29): boost-program-options-1.75.0-8.el9.x86 903 kB/s | 106 kB 00:00 2023-12-15T22:50:43.183 INFO:teuthology.orchestra.run.smithi072.stdout:(14/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 159 kB 00:00 2023-12-15T22:50:43.208 INFO:teuthology.orchestra.run.smithi072.stdout:(15/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 874 kB/s | 44 kB 00:00 2023-12-15T22:50:43.259 INFO:teuthology.orchestra.run.smithi072.stdout:(16/29): librdkafka-1.6.1-102.el9.x86_64.rpm 8.5 MB/s | 661 kB 00:00 2023-12-15T22:50:43.294 INFO:teuthology.orchestra.run.smithi072.stdout:(17/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2023-12-15T22:50:43.320 INFO:teuthology.orchestra.run.smithi072.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 685 kB/s | 41 kB 00:00 2023-12-15T22:50:43.662 INFO:teuthology.orchestra.run.smithi072.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 841 kB/s | 309 kB 00:00 2023-12-15T22:50:43.746 INFO:teuthology.orchestra.run.smithi072.stdout:(20/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 312 kB/s | 26 kB 00:00 2023-12-15T22:50:43.813 INFO:teuthology.orchestra.run.smithi072.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 725 kB/s | 49 kB 00:00 2023-12-15T22:50:43.874 INFO:teuthology.orchestra.run.smithi071.stdout:(26/29): librgw2-19.0.0-201.gcd750a7f.el9.x86_6 610 kB/s | 4.5 MB 00:07 2023-12-15T22:50:43.913 INFO:teuthology.orchestra.run.smithi072.stdout:(22/29): libarrow-9.0.0-10.el9.x86_64.rpm 7.4 MB/s | 4.4 MB 00:00 2023-12-15T22:50:43.939 INFO:teuthology.orchestra.run.smithi072.stdout:(23/29): libunwind-1.6.2-1.el9.x86_64.rpm 535 kB/s | 67 kB 00:00 2023-12-15T22:50:44.006 INFO:teuthology.orchestra.run.smithi072.stdout:(24/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.9 MB/s | 838 kB 00:00 2023-12-15T22:50:44.032 INFO:teuthology.orchestra.run.smithi072.stdout:(25/29): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2023-12-15T22:50:44.116 INFO:teuthology.orchestra.run.smithi072.stdout:(26/29): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2023-12-15T22:50:45.049 INFO:teuthology.orchestra.run.smithi071.stdout:(27/29): librados2-19.0.0-201.gcd750a7f.el9.x86 718 kB/s | 3.1 MB 00:04 2023-12-15T22:50:47.483 INFO:teuthology.orchestra.run.smithi072.stdout:(27/29): librados2-19.0.0-201.gcd750a7f.el9.x86 921 kB/s | 3.1 MB 00:03 2023-12-15T22:50:47.632 INFO:teuthology.orchestra.run.smithi071.stdout:(28/29): librbd1-19.0.0-201.gcd750a7f.el9.x86_6 823 kB/s | 3.0 MB 00:03 2023-12-15T22:50:48.558 INFO:teuthology.orchestra.run.smithi072.stdout:(28/29): librbd1-19.0.0-201.gcd750a7f.el9.x86_6 696 kB/s | 3.0 MB 00:04 2023-12-15T22:50:49.450 INFO:teuthology.orchestra.run.smithi072.stdout:(29/29): ceph-common-19.0.0-201.gcd750a7f.el9.x 783 kB/s | 17 MB 00:22 2023-12-15T22:50:49.454 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2023-12-15T22:50:49.454 INFO:teuthology.orchestra.run.smithi072.stdout:Total 2.4 MB/s | 54 MB 00:22 2023-12-15T22:50:49.455 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2023-12-15T22:50:49.491 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2023-12-15T22:50:49.491 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2023-12-15T22:50:49.944 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2023-12-15T22:50:49.944 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2023-12-15T22:50:50.824 INFO:teuthology.orchestra.run.smithi071.stdout:(29/29): ceph-common-19.0.0-201.gcd750a7f.el9.x 741 kB/s | 17 MB 00:23 2023-12-15T22:50:50.827 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-12-15T22:50:50.828 INFO:teuthology.orchestra.run.smithi071.stdout:Total 2.2 MB/s | 54 MB 00:24 2023-12-15T22:50:50.828 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-12-15T22:50:50.865 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-12-15T22:50:50.865 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-12-15T22:50:51.320 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-12-15T22:50:51.321 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-12-15T22:50:51.441 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2023-12-15T22:50:51.810 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2023-12-15T22:50:51.962 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : librados2-2:19.0.0-201.gcd750a7f.el9.x86_64 2/31 2023-12-15T22:50:52.054 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librados2-2:19.0.0-201.gcd750a7f.el9.x86_64 2/31 2023-12-15T22:50:52.151 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rados-2:19.0.0-201.gcd750a7f.el9.x86_64 3/31 2023-12-15T22:50:52.182 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libcephfs2-2:19.0.0-201.gcd750a7f.el9.x86_64 4/31 2023-12-15T22:50:52.252 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: libcephfs2-2:19.0.0-201.gcd750a7f.el9.x86_64 4/31 2023-12-15T22:50:52.286 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/31 2023-12-15T22:50:52.320 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 6/31 2023-12-15T22:50:52.361 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : liboath-2.6.7-2.el9.x86_64 7/31 2023-12-15T22:50:52.414 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-ceph-argparse-2:19.0.0-201.gcd750a7f.el9.x 8/31 2023-12-15T22:50:52.480 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cephfs-2:19.0.0-201.gcd750a7f.el9.x86_64 9/31 2023-12-15T22:50:52.506 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libradosstriper1-2:19.0.0-201.gcd750a7f.el9.x86_64 10/31 2023-12-15T22:50:52.593 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: libradosstriper1-2:19.0.0-201.gcd750a7f.el9.x86_64 10/31 2023-12-15T22:50:52.664 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : thrift-0.15.0-2.el9.x86_64 11/31 2023-12-15T22:50:52.680 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-12-15T22:50:52.771 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2023-12-15T22:50:52.808 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2023-12-15T22:50:52.853 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2023-12-15T22:50:52.884 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2023-12-15T22:50:52.913 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2023-12-15T22:50:52.946 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2023-12-15T22:50:53.023 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2023-12-15T22:50:53.092 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : librados2-2:19.0.0-201.gcd750a7f.el9.x86_64 2/31 2023-12-15T22:50:53.184 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados2-2:19.0.0-201.gcd750a7f.el9.x86_64 2/31 2023-12-15T22:50:53.258 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2023-12-15T22:50:53.272 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rados-2:19.0.0-201.gcd750a7f.el9.x86_64 3/31 2023-12-15T22:50:53.286 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : librbd1-2:19.0.0-201.gcd750a7f.el9.x86_64 19/31 2023-12-15T22:50:53.310 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephfs2-2:19.0.0-201.gcd750a7f.el9.x86_64 4/31 2023-12-15T22:50:53.383 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librbd1-2:19.0.0-201.gcd750a7f.el9.x86_64 19/31 2023-12-15T22:50:53.389 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephfs2-2:19.0.0-201.gcd750a7f.el9.x86_64 4/31 2023-12-15T22:50:53.416 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rbd-2:19.0.0-201.gcd750a7f.el9.x86_64 20/31 2023-12-15T22:50:53.422 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/31 2023-12-15T22:50:53.448 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 6/31 2023-12-15T22:50:53.454 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2023-12-15T22:50:53.486 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : liboath-2.6.7-2.el9.x86_64 7/31 2023-12-15T22:50:53.530 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ceph-argparse-2:19.0.0-201.gcd750a7f.el9.x 8/31 2023-12-15T22:50:53.596 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cephfs-2:19.0.0-201.gcd750a7f.el9.x86_64 9/31 2023-12-15T22:50:53.622 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libradosstriper1-2:19.0.0-201.gcd750a7f.el9.x86_64 10/31 2023-12-15T22:50:53.693 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libradosstriper1-2:19.0.0-201.gcd750a7f.el9.x86_64 10/31 2023-12-15T22:50:53.728 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : thrift-0.15.0-2.el9.x86_64 11/31 2023-12-15T22:50:53.837 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2023-12-15T22:50:53.872 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2023-12-15T22:50:53.881 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2023-12-15T22:50:53.906 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librgw2-2:19.0.0-201.gcd750a7f.el9.x86_64 23/31 2023-12-15T22:50:53.913 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2023-12-15T22:50:53.953 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2023-12-15T22:50:53.960 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librgw2-2:19.0.0-201.gcd750a7f.el9.x86_64 23/31 2023-12-15T22:50:53.982 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2023-12-15T22:50:54.014 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2023-12-15T22:50:54.028 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rgw-2:19.0.0-201.gcd750a7f.el9.x86_64 24/31 2023-12-15T22:50:54.060 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-ceph-common-2:19.0.0-201.gcd750a7f.el9.x86 25/31 2023-12-15T22:50:54.328 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2023-12-15T22:50:54.361 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : librbd1-2:19.0.0-201.gcd750a7f.el9.x86_64 19/31 2023-12-15T22:50:54.447 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librbd1-2:19.0.0-201.gcd750a7f.el9.x86_64 19/31 2023-12-15T22:50:54.483 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rbd-2:19.0.0-201.gcd750a7f.el9.x86_64 20/31 2023-12-15T22:50:54.518 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2023-12-15T22:50:54.951 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2023-12-15T22:50:54.980 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librgw2-2:19.0.0-201.gcd750a7f.el9.x86_64 23/31 2023-12-15T22:50:55.034 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librgw2-2:19.0.0-201.gcd750a7f.el9.x86_64 23/31 2023-12-15T22:50:55.103 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rgw-2:19.0.0-201.gcd750a7f.el9.x86_64 24/31 2023-12-15T22:50:55.133 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ceph-common-2:19.0.0-201.gcd750a7f.el9.x86 25/31 2023-12-15T22:50:56.131 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-common-2:19.0.0-201.gcd750a7f.el9.x86_64 26/31 2023-12-15T22:50:56.171 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-common-2:19.0.0-201.gcd750a7f.el9.x86_64 26/31 2023-12-15T22:50:56.890 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-common-2:19.0.0-201.gcd750a7f.el9.x86_64 26/31 2023-12-15T22:50:57.155 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-common-2:19.0.0-201.gcd750a7f.el9.x86_64 26/31 2023-12-15T22:50:57.180 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-base-2:19.0.0-201.gcd750a7f.el9.x86_64 27/31 2023-12-15T22:50:57.196 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-common-2:19.0.0-201.gcd750a7f.el9.x86_64 26/31 2023-12-15T22:50:57.301 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-base-2:19.0.0-201.gcd750a7f.el9.x86_64 27/31 2023-12-15T22:50:57.301 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-15T22:50:57.301 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-15T22:50:57.301 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:50:57.332 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-selinux-2:19.0.0-201.gcd750a7f.el9.x86_64 28/31 2023-12-15T22:50:57.925 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-common-2:19.0.0-201.gcd750a7f.el9.x86_64 26/31 2023-12-15T22:50:57.957 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-base-2:19.0.0-201.gcd750a7f.el9.x86_64 27/31 2023-12-15T22:50:58.044 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-base-2:19.0.0-201.gcd750a7f.el9.x86_64 27/31 2023-12-15T22:50:58.044 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-15T22:50:58.045 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-15T22:50:58.045 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:50:58.083 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-selinux-2:19.0.0-201.gcd750a7f.el9.x86_64 28/31 2023-12-15T22:51:07.518 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-selinux-2:19.0.0-201.gcd750a7f.el9.x86_64 28/31 2023-12-15T22:51:07.518 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /sys 2023-12-15T22:51:07.518 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /proc 2023-12-15T22:51:07.518 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /mnt 2023-12-15T22:51:07.519 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /var/tmp 2023-12-15T22:51:07.519 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /home 2023-12-15T22:51:07.519 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /root 2023-12-15T22:51:07.519 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /tmp 2023-12-15T22:51:07.519 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:51:08.015 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-selinux-2:19.0.0-201.gcd750a7f.el9.x86_64 28/31 2023-12-15T22:51:08.015 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /sys 2023-12-15T22:51:08.015 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /proc 2023-12-15T22:51:08.015 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /mnt 2023-12-15T22:51:08.015 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /var/tmp 2023-12-15T22:51:08.016 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /home 2023-12-15T22:51:08.016 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /root 2023-12-15T22:51:08.016 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /tmp 2023-12-15T22:51:08.016 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:51:08.383 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-radosgw-2:19.0.0-201.gcd750a7f.el9.x86_64 29/31 2023-12-15T22:51:08.407 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-201.gcd750a7f.el9.x86_64 29/31 2023-12-15T22:51:08.407 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-15T22:51:08.407 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-15T22:51:08.407 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-15T22:51:08.407 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-15T22:51:08.407 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:51:08.409 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-15T22:51:08.494 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-15T22:51:08.494 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-15T22:51:08.871 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-radosgw-2:19.0.0-201.gcd750a7f.el9.x86_64 29/31 2023-12-15T22:51:08.901 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-201.gcd750a7f.el9.x86_64 29/31 2023-12-15T22:51:08.901 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-15T22:51:08.901 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-15T22:51:08.901 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-15T22:51:08.902 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-15T22:51:08.902 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:51:08.903 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-15T22:51:08.994 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-15T22:51:08.995 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-15T22:51:09.628 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-15T22:51:09.629 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-base-2:19.0.0-201.gcd750a7f.el9.x86_64 1/31 2023-12-15T22:51:09.629 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-common-2:19.0.0-201.gcd750a7f.el9.x86_64 2/31 2023-12-15T22:51:09.629 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-radosgw-2:19.0.0-201.gcd750a7f.el9.x86_64 3/31 2023-12-15T22:51:09.629 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-selinux-2:19.0.0-201.gcd750a7f.el9.x86_64 4/31 2023-12-15T22:51:09.629 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephfs2-2:19.0.0-201.gcd750a7f.el9.x86_64 5/31 2023-12-15T22:51:09.629 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libradosstriper1-2:19.0.0-201.gcd750a7f.el9.x86_64 6/31 2023-12-15T22:51:09.629 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librgw2-2:19.0.0-201.gcd750a7f.el9.x86_64 7/31 2023-12-15T22:51:09.629 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ceph-argparse-2:19.0.0-201.gcd750a7f.el9.x 8/31 2023-12-15T22:51:09.629 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ceph-common-2:19.0.0-201.gcd750a7f.el9.x86 9/31 2023-12-15T22:51:09.629 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cephfs-2:19.0.0-201.gcd750a7f.el9.x86_64 10/31 2023-12-15T22:51:09.629 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rados-2:19.0.0-201.gcd750a7f.el9.x86_64 11/31 2023-12-15T22:51:09.629 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rbd-2:19.0.0-201.gcd750a7f.el9.x86_64 12/31 2023-12-15T22:51:09.629 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rgw-2:19.0.0-201.gcd750a7f.el9.x86_64 13/31 2023-12-15T22:51:09.629 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2023-12-15T22:51:09.629 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2023-12-15T22:51:09.630 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2023-12-15T22:51:09.630 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2023-12-15T22:51:09.630 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2023-12-15T22:51:09.630 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2023-12-15T22:51:09.630 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2023-12-15T22:51:09.630 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2023-12-15T22:51:09.630 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2023-12-15T22:51:09.630 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2023-12-15T22:51:09.630 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2023-12-15T22:51:09.630 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2023-12-15T22:51:09.630 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2023-12-15T22:51:09.630 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2023-12-15T22:51:09.630 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados2-2:19.0.0-201.gcd750a7f.el9.x86_64 28/31 2023-12-15T22:51:09.631 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2023-12-15T22:51:09.631 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librbd1-2:19.0.0-201.gcd750a7f.el9.x86_64 30/31 2023-12-15T22:51:10.410 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-15T22:51:10.410 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-base-2:19.0.0-201.gcd750a7f.el9.x86_64 1/31 2023-12-15T22:51:10.410 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-common-2:19.0.0-201.gcd750a7f.el9.x86_64 2/31 2023-12-15T22:51:10.410 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-radosgw-2:19.0.0-201.gcd750a7f.el9.x86_64 3/31 2023-12-15T22:51:10.410 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-selinux-2:19.0.0-201.gcd750a7f.el9.x86_64 4/31 2023-12-15T22:51:10.410 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libcephfs2-2:19.0.0-201.gcd750a7f.el9.x86_64 5/31 2023-12-15T22:51:10.410 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libradosstriper1-2:19.0.0-201.gcd750a7f.el9.x86_64 6/31 2023-12-15T22:51:10.410 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librgw2-2:19.0.0-201.gcd750a7f.el9.x86_64 7/31 2023-12-15T22:51:10.410 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-ceph-argparse-2:19.0.0-201.gcd750a7f.el9.x 8/31 2023-12-15T22:51:10.410 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-ceph-common-2:19.0.0-201.gcd750a7f.el9.x86 9/31 2023-12-15T22:51:10.410 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cephfs-2:19.0.0-201.gcd750a7f.el9.x86_64 10/31 2023-12-15T22:51:10.410 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rados-2:19.0.0-201.gcd750a7f.el9.x86_64 11/31 2023-12-15T22:51:10.411 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rbd-2:19.0.0-201.gcd750a7f.el9.x86_64 12/31 2023-12-15T22:51:10.411 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rgw-2:19.0.0-201.gcd750a7f.el9.x86_64 13/31 2023-12-15T22:51:10.411 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2023-12-15T22:51:10.411 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2023-12-15T22:51:10.411 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2023-12-15T22:51:10.411 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2023-12-15T22:51:10.411 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2023-12-15T22:51:10.411 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2023-12-15T22:51:10.411 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2023-12-15T22:51:10.411 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2023-12-15T22:51:10.411 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2023-12-15T22:51:10.411 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2023-12-15T22:51:10.411 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2023-12-15T22:51:10.411 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2023-12-15T22:51:10.411 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2023-12-15T22:51:10.411 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2023-12-15T22:51:10.412 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librados2-2:19.0.0-201.gcd750a7f.el9.x86_64 28/31 2023-12-15T22:51:10.412 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2023-12-15T22:51:10.412 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librbd1-2:19.0.0-201.gcd750a7f.el9.x86_64 30/31 2023-12-15T22:51:10.831 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2023-12-15T22:51:10.831 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:51:10.832 INFO:teuthology.orchestra.run.smithi071.stdout:Upgraded: 2023-12-15T22:51:10.832 INFO:teuthology.orchestra.run.smithi071.stdout: librados2-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:51:10.832 INFO:teuthology.orchestra.run.smithi071.stdout: librbd1-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:51:10.832 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-12-15T22:51:10.832 INFO:teuthology.orchestra.run.smithi071.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-12-15T22:51:10.832 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-base-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:51:10.832 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-common-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:51:10.832 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-radosgw-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:51:10.832 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-selinux-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:51:10.832 INFO:teuthology.orchestra.run.smithi071.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-15T22:51:10.832 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow-9.0.0-10.el9.x86_64 2023-12-15T22:51:10.832 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow-doc-9.0.0-10.el9.noarch 2023-12-15T22:51:10.832 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs2-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:51:10.832 INFO:teuthology.orchestra.run.smithi071.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-15T22:51:10.832 INFO:teuthology.orchestra.run.smithi071.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-15T22:51:10.832 INFO:teuthology.orchestra.run.smithi071.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-15T22:51:10.833 INFO:teuthology.orchestra.run.smithi071.stdout: libradosstriper1-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:51:10.833 INFO:teuthology.orchestra.run.smithi071.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-15T22:51:10.833 INFO:teuthology.orchestra.run.smithi071.stdout: librgw2-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:51:10.833 INFO:teuthology.orchestra.run.smithi071.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-15T22:51:10.833 INFO:teuthology.orchestra.run.smithi071.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-15T22:51:10.833 INFO:teuthology.orchestra.run.smithi071.stdout: parquet-libs-9.0.0-10.el9.x86_64 2023-12-15T22:51:10.833 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-argparse-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:51:10.833 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-common-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:51:10.833 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cephfs-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:51:10.833 INFO:teuthology.orchestra.run.smithi071.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-15T22:51:10.833 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rados-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:51:10.833 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rbd-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:51:10.833 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rgw-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:51:10.833 INFO:teuthology.orchestra.run.smithi071.stdout: re2-1:20211101-3.el9.x86_64 2023-12-15T22:51:10.834 INFO:teuthology.orchestra.run.smithi071.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-15T22:51:10.834 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:51:10.834 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-12-15T22:51:11.224 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-test 2023-12-15T22:51:11.553 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2023-12-15T22:51:11.554 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:51:11.554 INFO:teuthology.orchestra.run.smithi072.stdout:Upgraded: 2023-12-15T22:51:11.554 INFO:teuthology.orchestra.run.smithi072.stdout: librados2-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:51:11.554 INFO:teuthology.orchestra.run.smithi072.stdout: librbd1-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:51:11.554 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2023-12-15T22:51:11.554 INFO:teuthology.orchestra.run.smithi072.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-12-15T22:51:11.554 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-base-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:51:11.555 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-common-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:51:11.555 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-radosgw-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:51:11.555 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-selinux-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:51:11.555 INFO:teuthology.orchestra.run.smithi072.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-15T22:51:11.555 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow-9.0.0-10.el9.x86_64 2023-12-15T22:51:11.555 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow-doc-9.0.0-10.el9.noarch 2023-12-15T22:51:11.555 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs2-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:51:11.555 INFO:teuthology.orchestra.run.smithi072.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-15T22:51:11.555 INFO:teuthology.orchestra.run.smithi072.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-15T22:51:11.555 INFO:teuthology.orchestra.run.smithi072.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-15T22:51:11.555 INFO:teuthology.orchestra.run.smithi072.stdout: libradosstriper1-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:51:11.555 INFO:teuthology.orchestra.run.smithi072.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-15T22:51:11.555 INFO:teuthology.orchestra.run.smithi072.stdout: librgw2-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:51:11.555 INFO:teuthology.orchestra.run.smithi072.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-15T22:51:11.555 INFO:teuthology.orchestra.run.smithi072.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-15T22:51:11.555 INFO:teuthology.orchestra.run.smithi072.stdout: parquet-libs-9.0.0-10.el9.x86_64 2023-12-15T22:51:11.556 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-argparse-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:51:11.556 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-common-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:51:11.556 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cephfs-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:51:11.556 INFO:teuthology.orchestra.run.smithi072.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-15T22:51:11.556 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rados-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:51:11.556 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rbd-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:51:11.556 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rgw-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:51:11.556 INFO:teuthology.orchestra.run.smithi072.stdout: re2-1:20211101-3.el9.x86_64 2023-12-15T22:51:11.556 INFO:teuthology.orchestra.run.smithi072.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-15T22:51:11.556 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:51:11.556 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-12-15T22:51:11.783 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:47 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:51:11.925 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-test 2023-12-15T22:51:11.927 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-12-15T22:51:11.928 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-12-15T22:51:11.928 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-12-15T22:51:11.928 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-12-15T22:51:11.928 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-12-15T22:51:11.928 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-test x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 118 M 2023-12-15T22:51:11.928 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2023-12-15T22:51:11.928 INFO:teuthology.orchestra.run.smithi071.stdout: cryptopp x86_64 8.6.0-1.el9 epel 1.3 M 2023-12-15T22:51:11.928 INFO:teuthology.orchestra.run.smithi071.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-15T22:51:11.929 INFO:teuthology.orchestra.run.smithi071.stdout: libasan x86_64 11.4.1-2.3.el9 CentOS-AppStream 418 k 2023-12-15T22:51:11.929 INFO:teuthology.orchestra.run.smithi071.stdout: libcephsqlite x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 159 k 2023-12-15T22:51:11.929 INFO:teuthology.orchestra.run.smithi071.stdout: libubsan x86_64 11.4.1-2.3.el9 CentOS-AppStream 186 k 2023-12-15T22:51:11.929 INFO:teuthology.orchestra.run.smithi071.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-15T22:51:11.929 INFO:teuthology.orchestra.run.smithi071.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-15T22:51:11.929 INFO:teuthology.orchestra.run.smithi071.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-15T22:51:11.929 INFO:teuthology.orchestra.run.smithi071.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2023-12-15T22:51:11.929 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:51:11.929 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-12-15T22:51:11.929 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-12-15T22:51:11.929 INFO:teuthology.orchestra.run.smithi071.stdout:Install 10 Packages 2023-12-15T22:51:11.929 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:51:11.930 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 121 M 2023-12-15T22:51:11.930 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 1.1 G 2023-12-15T22:51:11.930 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-12-15T22:51:12.345 INFO:teuthology.orchestra.run.smithi071.stdout:(1/10): jq-1.6-16.el9.x86_64.rpm 1.0 MB/s | 186 kB 00:00 2023-12-15T22:51:12.398 INFO:teuthology.orchestra.run.smithi071.stdout:(2/10): libasan-11.4.1-2.3.el9.x86_64.rpm 8.0 MB/s | 418 kB 00:00 2023-12-15T22:51:12.431 INFO:teuthology.orchestra.run.smithi071.stdout:(3/10): libubsan-11.4.1-2.3.el9.x86_64.rpm 5.5 MB/s | 186 kB 00:00 2023-12-15T22:51:12.465 INFO:teuthology.orchestra.run.smithi071.stdout:(4/10): oniguruma-6.9.6-1.el9.5.x86_64.rpm 6.4 MB/s | 217 kB 00:00 2023-12-15T22:51:12.489 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:00:48 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:51:12.498 INFO:teuthology.orchestra.run.smithi071.stdout:(5/10): socat-1.7.4.1-5.el9.x86_64.rpm 8.8 MB/s | 300 kB 00:00 2023-12-15T22:51:12.533 INFO:teuthology.orchestra.run.smithi071.stdout:(6/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2023-12-15T22:51:12.641 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-12-15T22:51:12.642 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-12-15T22:51:12.642 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2023-12-15T22:51:12.642 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-12-15T22:51:12.642 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2023-12-15T22:51:12.642 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-test x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 118 M 2023-12-15T22:51:12.642 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2023-12-15T22:51:12.642 INFO:teuthology.orchestra.run.smithi072.stdout: cryptopp x86_64 8.6.0-1.el9 epel 1.3 M 2023-12-15T22:51:12.642 INFO:teuthology.orchestra.run.smithi072.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-15T22:51:12.643 INFO:teuthology.orchestra.run.smithi072.stdout: libasan x86_64 11.4.1-2.3.el9 CentOS-AppStream 418 k 2023-12-15T22:51:12.643 INFO:teuthology.orchestra.run.smithi072.stdout: libcephsqlite x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 159 k 2023-12-15T22:51:12.643 INFO:teuthology.orchestra.run.smithi072.stdout: libubsan x86_64 11.4.1-2.3.el9 CentOS-AppStream 186 k 2023-12-15T22:51:12.643 INFO:teuthology.orchestra.run.smithi072.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-15T22:51:12.643 INFO:teuthology.orchestra.run.smithi072.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-15T22:51:12.643 INFO:teuthology.orchestra.run.smithi072.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-15T22:51:12.643 INFO:teuthology.orchestra.run.smithi072.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2023-12-15T22:51:12.643 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:51:12.643 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2023-12-15T22:51:12.643 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-12-15T22:51:12.643 INFO:teuthology.orchestra.run.smithi072.stdout:Install 10 Packages 2023-12-15T22:51:12.643 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:51:12.644 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 121 M 2023-12-15T22:51:12.644 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 1.1 G 2023-12-15T22:51:12.644 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2023-12-15T22:51:12.799 INFO:teuthology.orchestra.run.smithi071.stdout:(7/10): libcephsqlite-19.0.0-201.gcd750a7f.el9. 250 kB/s | 159 kB 00:00 2023-12-15T22:51:12.950 INFO:teuthology.orchestra.run.smithi072.stdout:(1/10): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 186 kB 00:00 2023-12-15T22:51:13.009 INFO:teuthology.orchestra.run.smithi072.stdout:(2/10): libasan-11.4.1-2.3.el9.x86_64.rpm 7.0 MB/s | 418 kB 00:00 2023-12-15T22:51:13.042 INFO:teuthology.orchestra.run.smithi072.stdout:(3/10): libubsan-11.4.1-2.3.el9.x86_64.rpm 5.6 MB/s | 186 kB 00:00 2023-12-15T22:51:13.076 INFO:teuthology.orchestra.run.smithi072.stdout:(4/10): oniguruma-6.9.6-1.el9.5.x86_64.rpm 6.3 MB/s | 217 kB 00:00 2023-12-15T22:51:13.110 INFO:teuthology.orchestra.run.smithi072.stdout:(5/10): socat-1.7.4.1-5.el9.x86_64.rpm 8.7 MB/s | 300 kB 00:00 2023-12-15T22:51:13.144 INFO:teuthology.orchestra.run.smithi072.stdout:(6/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2023-12-15T22:51:13.328 INFO:teuthology.orchestra.run.smithi072.stdout:(7/10): cryptopp-8.6.0-1.el9.x86_64.rpm 6.8 MB/s | 1.3 MB 00:00 2023-12-15T22:51:13.370 INFO:teuthology.orchestra.run.smithi072.stdout:(8/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 2.8 MB/s | 122 kB 00:00 2023-12-15T22:51:13.566 INFO:teuthology.orchestra.run.smithi071.stdout:(8/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 159 kB/s | 122 kB 00:00 2023-12-15T22:51:13.587 INFO:teuthology.orchestra.run.smithi072.stdout:(9/10): libcephsqlite-19.0.0-201.gcd750a7f.el9. 201 kB/s | 159 kB 00:00 2023-12-15T22:51:13.791 INFO:teuthology.orchestra.run.smithi071.stdout:(9/10): cryptopp-8.6.0-1.el9.x86_64.rpm 1.0 MB/s | 1.3 MB 00:01 2023-12-15T22:52:38.301 INFO:teuthology.orchestra.run.smithi071.stdout:(10/10): ceph-test-19.0.0-201.gcd750a7f.el9.x86 1.4 MB/s | 118 MB 01:26 2023-12-15T22:52:38.302 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-12-15T22:52:38.302 INFO:teuthology.orchestra.run.smithi071.stdout:Total 1.4 MB/s | 121 MB 01:26 2023-12-15T22:52:38.302 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-12-15T22:52:38.320 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-12-15T22:52:38.320 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-12-15T22:52:39.132 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-12-15T22:52:39.133 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-12-15T22:52:40.279 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-12-15T22:52:40.358 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2023-12-15T22:52:40.407 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : cryptopp-8.6.0-1.el9.x86_64 2/10 2023-12-15T22:52:40.463 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/10 2023-12-15T22:52:40.501 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/10 2023-12-15T22:52:40.533 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 5/10 2023-12-15T22:52:40.566 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : jq-1.6-16.el9.x86_64 6/10 2023-12-15T22:52:40.594 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libubsan-11.4.1-2.3.el9.x86_64 7/10 2023-12-15T22:52:40.636 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libasan-11.4.1-2.3.el9.x86_64 8/10 2023-12-15T22:52:40.670 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephsqlite-2:19.0.0-201.gcd750a7f.el9.x86_64 9/10 2023-12-15T22:52:56.732 INFO:teuthology.orchestra.run.smithi072.stdout:(10/10): ceph-test-19.0.0-201.gcd750a7f.el9.x86 1.1 MB/s | 118 MB 01:43 2023-12-15T22:52:56.734 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2023-12-15T22:52:56.735 INFO:teuthology.orchestra.run.smithi072.stdout:Total 1.2 MB/s | 121 MB 01:44 2023-12-15T22:52:56.735 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2023-12-15T22:52:56.756 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2023-12-15T22:52:56.756 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2023-12-15T22:52:57.134 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephsqlite-2:19.0.0-201.gcd750a7f.el9.x86_64 9/10 2023-12-15T22:52:57.207 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-test-2:19.0.0-201.gcd750a7f.el9.x86_64 10/10 2023-12-15T22:52:57.577 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2023-12-15T22:52:57.577 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2023-12-15T22:52:58.755 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2023-12-15T22:52:58.832 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2023-12-15T22:52:58.866 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : cryptopp-8.6.0-1.el9.x86_64 2/10 2023-12-15T22:52:58.923 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/10 2023-12-15T22:52:58.963 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/10 2023-12-15T22:52:58.994 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 5/10 2023-12-15T22:52:59.027 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : jq-1.6-16.el9.x86_64 6/10 2023-12-15T22:52:59.056 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libubsan-11.4.1-2.3.el9.x86_64 7/10 2023-12-15T22:52:59.097 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libasan-11.4.1-2.3.el9.x86_64 8/10 2023-12-15T22:52:59.126 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libcephsqlite-2:19.0.0-201.gcd750a7f.el9.x86_64 9/10 2023-12-15T22:53:05.336 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-test-2:19.0.0-201.gcd750a7f.el9.x86_64 10/10 2023-12-15T22:53:05.336 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-test-2:19.0.0-201.gcd750a7f.el9.x86_64 1/10 2023-12-15T22:53:05.336 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephsqlite-2:19.0.0-201.gcd750a7f.el9.x86_64 2/10 2023-12-15T22:53:05.336 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : jq-1.6-16.el9.x86_64 3/10 2023-12-15T22:53:05.336 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libasan-11.4.1-2.3.el9.x86_64 4/10 2023-12-15T22:53:05.336 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libubsan-11.4.1-2.3.el9.x86_64 5/10 2023-12-15T22:53:05.336 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/10 2023-12-15T22:53:05.336 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 7/10 2023-12-15T22:53:05.336 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/10 2023-12-15T22:53:05.336 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : cryptopp-8.6.0-1.el9.x86_64 9/10 2023-12-15T22:53:05.811 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2023-12-15T22:53:05.811 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:53:05.811 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-12-15T22:53:05.812 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-test-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:53:05.812 INFO:teuthology.orchestra.run.smithi071.stdout: cryptopp-8.6.0-1.el9.x86_64 2023-12-15T22:53:05.812 INFO:teuthology.orchestra.run.smithi071.stdout: jq-1.6-16.el9.x86_64 2023-12-15T22:53:05.812 INFO:teuthology.orchestra.run.smithi071.stdout: libasan-11.4.1-2.3.el9.x86_64 2023-12-15T22:53:05.812 INFO:teuthology.orchestra.run.smithi071.stdout: libcephsqlite-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:53:05.812 INFO:teuthology.orchestra.run.smithi071.stdout: libubsan-11.4.1-2.3.el9.x86_64 2023-12-15T22:53:05.812 INFO:teuthology.orchestra.run.smithi071.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-12-15T22:53:05.812 INFO:teuthology.orchestra.run.smithi071.stdout: socat-1.7.4.1-5.el9.x86_64 2023-12-15T22:53:05.812 INFO:teuthology.orchestra.run.smithi071.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-12-15T22:53:05.812 INFO:teuthology.orchestra.run.smithi071.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2023-12-15T22:53:05.812 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:53:05.812 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-12-15T22:53:06.144 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph 2023-12-15T22:53:06.698 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:42 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:53:06.846 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-12-15T22:53:06.849 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================== 2023-12-15T22:53:06.849 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-12-15T22:53:06.849 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================== 2023-12-15T22:53:06.849 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-12-15T22:53:06.849 INFO:teuthology.orchestra.run.smithi071.stdout: ceph x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 6.5 k 2023-12-15T22:53:06.849 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2023-12-15T22:53:06.849 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mds x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 2.0 M 2023-12-15T22:53:06.849 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 1.5 M 2023-12-15T22:53:06.849 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-modules-core noarch 2:19.0.0-201.gcd750a7f.el9 ceph-noarch 243 k 2023-12-15T22:53:06.849 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mon x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 8.9 M 2023-12-15T22:53:06.849 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-osd x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 117 M 2023-12-15T22:53:06.849 INFO:teuthology.orchestra.run.smithi071.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-12-15T22:53:06.850 INFO:teuthology.orchestra.run.smithi071.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-12-15T22:53:06.850 INFO:teuthology.orchestra.run.smithi071.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-15T22:53:06.850 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-15T22:53:06.850 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-15T22:53:06.850 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-15T22:53:06.850 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-15T22:53:06.850 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-15T22:53:06.850 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-15T22:53:06.850 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-15T22:53:06.850 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-15T22:53:06.850 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-15T22:53:06.850 INFO:teuthology.orchestra.run.smithi071.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-15T22:53:06.850 INFO:teuthology.orchestra.run.smithi071.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-15T22:53:06.851 INFO:teuthology.orchestra.run.smithi071.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-15T22:53:06.851 INFO:teuthology.orchestra.run.smithi071.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-15T22:53:06.851 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-15T22:53:06.851 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-15T22:53:06.851 INFO:teuthology.orchestra.run.smithi071.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-15T22:53:06.851 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-15T22:53:06.851 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-15T22:53:06.851 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-15T22:53:06.851 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-15T22:53:06.851 INFO:teuthology.orchestra.run.smithi071.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-15T22:53:06.851 INFO:teuthology.orchestra.run.smithi071.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-15T22:53:06.851 INFO:teuthology.orchestra.run.smithi071.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-15T22:53:06.851 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-15T22:53:06.851 INFO:teuthology.orchestra.run.smithi071.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-15T22:53:06.851 INFO:teuthology.orchestra.run.smithi071.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-15T22:53:06.852 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:53:06.852 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-12-15T22:53:06.852 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================== 2023-12-15T22:53:06.852 INFO:teuthology.orchestra.run.smithi071.stdout:Install 35 Packages 2023-12-15T22:53:06.852 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:53:06.852 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 134 M 2023-12-15T22:53:06.852 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 1.4 G 2023-12-15T22:53:06.852 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-12-15T22:53:07.282 INFO:teuthology.orchestra.run.smithi071.stdout:(1/35): ceph-19.0.0-201.gcd750a7f.el9.x86_64.rp 19 kB/s | 6.5 kB 00:00 2023-12-15T22:53:08.082 INFO:teuthology.orchestra.run.smithi071.stdout:(2/35): ceph-mgr-19.0.0-201.gcd750a7f.el9.x86_6 1.3 MB/s | 1.5 MB 00:01 2023-12-15T22:53:08.141 INFO:teuthology.orchestra.run.smithi071.stdout:(3/35): ceph-mds-19.0.0-201.gcd750a7f.el9.x86_6 1.6 MB/s | 2.0 MB 00:01 2023-12-15T22:53:08.249 INFO:teuthology.orchestra.run.smithi071.stdout:(4/35): ceph-mgr-modules-core-19.0.0-201.gcd750 2.2 MB/s | 243 kB 00:00 2023-12-15T22:53:08.358 INFO:teuthology.orchestra.run.smithi071.stdout:(5/35): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2023-12-15T22:53:08.391 INFO:teuthology.orchestra.run.smithi071.stdout:(6/35): python3-mako-1.1.4-6.el9.noarch.rpm 4.3 MB/s | 144 kB 00:00 2023-12-15T22:53:08.425 INFO:teuthology.orchestra.run.smithi071.stdout:(7/35): python3-markupsafe-1.1.1-12.el9.x86_64. 947 kB/s | 32 kB 00:00 2023-12-15T22:53:08.459 INFO:teuthology.orchestra.run.smithi071.stdout:(8/35): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2023-12-15T22:53:08.492 INFO:teuthology.orchestra.run.smithi071.stdout:(9/35): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2023-12-15T22:53:08.534 INFO:teuthology.orchestra.run.smithi071.stdout:(10/35): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2023-12-15T22:53:08.601 INFO:teuthology.orchestra.run.smithi071.stdout:(11/35): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2023-12-15T22:53:08.635 INFO:teuthology.orchestra.run.smithi071.stdout:(12/35): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2023-12-15T22:53:08.668 INFO:teuthology.orchestra.run.smithi071.stdout:(13/35): python3-pycparser-2.20-6.el9.noarch.rp 3.7 MB/s | 124 kB 00:00 2023-12-15T22:53:08.702 INFO:teuthology.orchestra.run.smithi071.stdout:(14/35): python3-requests-2.25.1-7.el9.noarch.r 3.3 MB/s | 113 kB 00:00 2023-12-15T22:53:08.736 INFO:teuthology.orchestra.run.smithi071.stdout:(15/35): python3-urllib3-1.26.5-4.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2023-12-15T22:53:08.853 INFO:teuthology.orchestra.run.smithi071.stdout:(16/35): luarocks-3.9.2-1.el9.noarch.rpm 1.3 MB/s | 151 kB 00:00 2023-12-15T22:53:08.895 INFO:teuthology.orchestra.run.smithi071.stdout:(17/35): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.0 MB/s | 43 kB 00:00 2023-12-15T22:53:08.953 INFO:teuthology.orchestra.run.smithi071.stdout:(18/35): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2023-12-15T22:53:09.029 INFO:teuthology.orchestra.run.smithi071.stdout:(19/35): python3-cherrypy-18.6.1-2.el9.noarch.r 4.7 MB/s | 358 kB 00:00 2023-12-15T22:53:09.071 INFO:teuthology.orchestra.run.smithi071.stdout:(20/35): python3-jaraco-8.2.1-3.el9.noarch.rpm 256 kB/s | 11 kB 00:00 2023-12-15T22:53:09.113 INFO:teuthology.orchestra.run.smithi071.stdout:(21/35): python3-jaraco-classes-3.2.1-5.el9.noa 423 kB/s | 18 kB 00:00 2023-12-15T22:53:09.154 INFO:teuthology.orchestra.run.smithi071.stdout:(22/35): python3-jaraco-collections-3.0.0-8.el9 558 kB/s | 23 kB 00:00 2023-12-15T22:53:09.197 INFO:teuthology.orchestra.run.smithi071.stdout:(23/35): python3-jaraco-functools-3.5.0-2.el9.n 465 kB/s | 19 kB 00:00 2023-12-15T22:53:09.239 INFO:teuthology.orchestra.run.smithi071.stdout:(24/35): python3-jaraco-text-3.2.0-6.el9.noarch 473 kB/s | 20 kB 00:00 2023-12-15T22:53:09.280 INFO:teuthology.orchestra.run.smithi071.stdout:(25/35): python3-logutils-0.3.5-21.el9.noarch.r 1.1 MB/s | 46 kB 00:00 2023-12-15T22:53:09.322 INFO:teuthology.orchestra.run.smithi071.stdout:(26/35): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2023-12-15T22:53:09.381 INFO:teuthology.orchestra.run.smithi071.stdout:(27/35): python3-pecan-1.4.2-3.el9.noarch.rpm 4.6 MB/s | 272 kB 00:00 2023-12-15T22:53:09.423 INFO:teuthology.orchestra.run.smithi071.stdout:(28/35): python3-portend-3.1.0-2.el9.noarch.rpm 396 kB/s | 16 kB 00:00 2023-12-15T22:53:09.465 INFO:teuthology.orchestra.run.smithi071.stdout:(29/35): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2023-12-15T22:53:09.499 INFO:teuthology.orchestra.run.smithi071.stdout:(30/35): python3-tempora-5.0.0-2.el9.noarch.rpm 1.0 MB/s | 36 kB 00:00 2023-12-15T22:53:09.557 INFO:teuthology.orchestra.run.smithi071.stdout:(31/35): python3-webob-1.8.7-6.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2023-12-15T22:53:09.616 INFO:teuthology.orchestra.run.smithi071.stdout:(32/35): python3-werkzeug-2.0.3-3.el9.1.noarch. 7.1 MB/s | 427 kB 00:00 2023-12-15T22:53:09.649 INFO:teuthology.orchestra.run.smithi071.stdout:(33/35): python3-zc-lockfile-2.0-10.el9.noarch. 605 kB/s | 20 kB 00:00 2023-12-15T22:53:10.758 INFO:teuthology.orchestra.run.smithi071.stdout:(34/35): ceph-mon-19.0.0-201.gcd750a7f.el9.x86_ 2.6 MB/s | 8.9 MB 00:03 2023-12-15T22:53:15.363 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: libcephsqlite-2:19.0.0-201.gcd750a7f.el9.x86_64 9/10 2023-12-15T22:53:15.439 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-test-2:19.0.0-201.gcd750a7f.el9.x86_64 10/10 2023-12-15T22:53:31.214 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-test-2:19.0.0-201.gcd750a7f.el9.x86_64 10/10 2023-12-15T22:53:31.215 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-test-2:19.0.0-201.gcd750a7f.el9.x86_64 1/10 2023-12-15T22:53:31.215 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libcephsqlite-2:19.0.0-201.gcd750a7f.el9.x86_64 2/10 2023-12-15T22:53:31.215 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : jq-1.6-16.el9.x86_64 3/10 2023-12-15T22:53:31.215 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libasan-11.4.1-2.3.el9.x86_64 4/10 2023-12-15T22:53:31.215 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libubsan-11.4.1-2.3.el9.x86_64 5/10 2023-12-15T22:53:31.215 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/10 2023-12-15T22:53:31.215 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 7/10 2023-12-15T22:53:31.215 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/10 2023-12-15T22:53:31.215 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : cryptopp-8.6.0-1.el9.x86_64 9/10 2023-12-15T22:53:31.663 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2023-12-15T22:53:31.663 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:53:31.663 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2023-12-15T22:53:31.663 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-test-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:53:31.663 INFO:teuthology.orchestra.run.smithi072.stdout: cryptopp-8.6.0-1.el9.x86_64 2023-12-15T22:53:31.663 INFO:teuthology.orchestra.run.smithi072.stdout: jq-1.6-16.el9.x86_64 2023-12-15T22:53:31.663 INFO:teuthology.orchestra.run.smithi072.stdout: libasan-11.4.1-2.3.el9.x86_64 2023-12-15T22:53:31.663 INFO:teuthology.orchestra.run.smithi072.stdout: libcephsqlite-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:53:31.663 INFO:teuthology.orchestra.run.smithi072.stdout: libubsan-11.4.1-2.3.el9.x86_64 2023-12-15T22:53:31.663 INFO:teuthology.orchestra.run.smithi072.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-12-15T22:53:31.664 INFO:teuthology.orchestra.run.smithi072.stdout: socat-1.7.4.1-5.el9.x86_64 2023-12-15T22:53:31.664 INFO:teuthology.orchestra.run.smithi072.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-12-15T22:53:31.664 INFO:teuthology.orchestra.run.smithi072.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2023-12-15T22:53:31.664 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:53:31.664 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-12-15T22:53:32.029 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph 2023-12-15T22:53:32.580 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:03:08 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:53:32.740 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-12-15T22:53:32.742 INFO:teuthology.orchestra.run.smithi072.stdout:======================================================================================== 2023-12-15T22:53:32.742 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2023-12-15T22:53:32.742 INFO:teuthology.orchestra.run.smithi072.stdout:======================================================================================== 2023-12-15T22:53:32.742 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2023-12-15T22:53:32.742 INFO:teuthology.orchestra.run.smithi072.stdout: ceph x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 6.5 k 2023-12-15T22:53:32.743 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2023-12-15T22:53:32.743 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mds x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 2.0 M 2023-12-15T22:53:32.743 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 1.5 M 2023-12-15T22:53:32.743 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-modules-core noarch 2:19.0.0-201.gcd750a7f.el9 ceph-noarch 243 k 2023-12-15T22:53:32.743 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mon x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 8.9 M 2023-12-15T22:53:32.743 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-osd x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 117 M 2023-12-15T22:53:32.743 INFO:teuthology.orchestra.run.smithi072.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-12-15T22:53:32.743 INFO:teuthology.orchestra.run.smithi072.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-12-15T22:53:32.743 INFO:teuthology.orchestra.run.smithi072.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-15T22:53:32.743 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-15T22:53:32.743 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-15T22:53:32.743 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-15T22:53:32.743 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-15T22:53:32.744 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-15T22:53:32.744 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-15T22:53:32.744 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-15T22:53:32.744 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-15T22:53:32.744 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-15T22:53:32.744 INFO:teuthology.orchestra.run.smithi072.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-15T22:53:32.744 INFO:teuthology.orchestra.run.smithi072.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-15T22:53:32.744 INFO:teuthology.orchestra.run.smithi072.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-15T22:53:32.744 INFO:teuthology.orchestra.run.smithi072.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-15T22:53:32.744 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-15T22:53:32.744 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-15T22:53:32.744 INFO:teuthology.orchestra.run.smithi072.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-15T22:53:32.744 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-15T22:53:32.744 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-15T22:53:32.745 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-15T22:53:32.745 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-15T22:53:32.745 INFO:teuthology.orchestra.run.smithi072.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-15T22:53:32.745 INFO:teuthology.orchestra.run.smithi072.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-15T22:53:32.745 INFO:teuthology.orchestra.run.smithi072.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-15T22:53:32.745 INFO:teuthology.orchestra.run.smithi072.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-15T22:53:32.745 INFO:teuthology.orchestra.run.smithi072.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-15T22:53:32.745 INFO:teuthology.orchestra.run.smithi072.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-15T22:53:32.745 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:53:32.745 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2023-12-15T22:53:32.745 INFO:teuthology.orchestra.run.smithi072.stdout:======================================================================================== 2023-12-15T22:53:32.745 INFO:teuthology.orchestra.run.smithi072.stdout:Install 35 Packages 2023-12-15T22:53:32.745 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:53:32.746 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 134 M 2023-12-15T22:53:32.746 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 1.4 G 2023-12-15T22:53:32.746 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2023-12-15T22:53:33.291 INFO:teuthology.orchestra.run.smithi072.stdout:(1/35): ceph-19.0.0-201.gcd750a7f.el9.x86_64.rp 17 kB/s | 6.5 kB 00:00 2023-12-15T22:53:35.458 INFO:teuthology.orchestra.run.smithi072.stdout:(2/35): ceph-mgr-19.0.0-201.gcd750a7f.el9.x86_6 583 kB/s | 1.5 MB 00:02 2023-12-15T22:53:37.692 INFO:teuthology.orchestra.run.smithi072.stdout:(3/35): ceph-mds-19.0.0-201.gcd750a7f.el9.x86_6 428 kB/s | 2.0 MB 00:04 2023-12-15T22:53:38.075 INFO:teuthology.orchestra.run.smithi072.stdout:(4/35): ceph-mgr-modules-core-19.0.0-201.gcd750 634 kB/s | 243 kB 00:00 2023-12-15T22:53:38.185 INFO:teuthology.orchestra.run.smithi072.stdout:(5/35): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2023-12-15T22:53:38.235 INFO:teuthology.orchestra.run.smithi072.stdout:(6/35): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2023-12-15T22:53:38.260 INFO:teuthology.orchestra.run.smithi072.stdout:(7/35): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2023-12-15T22:53:38.287 INFO:teuthology.orchestra.run.smithi072.stdout:(8/35): python3-pytz-2021.1-5.el9.noarch.rpm 1.7 MB/s | 47 kB 00:00 2023-12-15T22:53:38.321 INFO:teuthology.orchestra.run.smithi072.stdout:(9/35): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2023-12-15T22:53:38.363 INFO:teuthology.orchestra.run.smithi072.stdout:(10/35): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2023-12-15T22:53:38.430 INFO:teuthology.orchestra.run.smithi072.stdout:(11/35): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2023-12-15T22:53:38.473 INFO:teuthology.orchestra.run.smithi072.stdout:(12/35): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2023-12-15T22:53:38.548 INFO:teuthology.orchestra.run.smithi072.stdout:(13/35): python3-pycparser-2.20-6.el9.noarch.rp 1.6 MB/s | 124 kB 00:00 2023-12-15T22:53:38.582 INFO:teuthology.orchestra.run.smithi072.stdout:(14/35): python3-requests-2.25.1-7.el9.noarch.r 3.3 MB/s | 113 kB 00:00 2023-12-15T22:53:38.616 INFO:teuthology.orchestra.run.smithi072.stdout:(15/35): python3-urllib3-1.26.5-4.el9.noarch.rp 5.4 MB/s | 187 kB 00:00 2023-12-15T22:53:39.150 INFO:teuthology.orchestra.run.smithi072.stdout:(16/35): luarocks-3.9.2-1.el9.noarch.rpm 283 kB/s | 151 kB 00:00 2023-12-15T22:53:39.259 INFO:teuthology.orchestra.run.smithi072.stdout:(17/35): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 400 kB/s | 43 kB 00:00 2023-12-15T22:53:39.418 INFO:teuthology.orchestra.run.smithi072.stdout:(18/35): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2023-12-15T22:53:39.610 INFO:teuthology.orchestra.run.smithi072.stdout:(19/35): python3-cherrypy-18.6.1-2.el9.noarch.r 1.8 MB/s | 358 kB 00:00 2023-12-15T22:53:39.702 INFO:teuthology.orchestra.run.smithi072.stdout:(20/35): python3-jaraco-8.2.1-3.el9.noarch.rpm 116 kB/s | 11 kB 00:00 2023-12-15T22:53:39.794 INFO:teuthology.orchestra.run.smithi072.stdout:(21/35): python3-jaraco-classes-3.2.1-5.el9.noa 192 kB/s | 18 kB 00:00 2023-12-15T22:53:39.886 INFO:teuthology.orchestra.run.smithi072.stdout:(22/35): python3-jaraco-collections-3.0.0-8.el9 252 kB/s | 23 kB 00:00 2023-12-15T22:53:39.980 INFO:teuthology.orchestra.run.smithi072.stdout:(23/35): python3-jaraco-functools-3.5.0-2.el9.n 208 kB/s | 19 kB 00:00 2023-12-15T22:53:40.081 INFO:teuthology.orchestra.run.smithi072.stdout:(24/35): python3-jaraco-text-3.2.0-6.el9.noarch 196 kB/s | 20 kB 00:00 2023-12-15T22:53:40.181 INFO:teuthology.orchestra.run.smithi072.stdout:(25/35): python3-logutils-0.3.5-21.el9.noarch.r 462 kB/s | 46 kB 00:00 2023-12-15T22:53:40.290 INFO:teuthology.orchestra.run.smithi072.stdout:(26/35): python3-more-itertools-8.12.0-2.el9.no 727 kB/s | 79 kB 00:00 2023-12-15T22:53:40.416 INFO:teuthology.orchestra.run.smithi072.stdout:(27/35): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2023-12-15T22:53:40.508 INFO:teuthology.orchestra.run.smithi072.stdout:(28/35): python3-portend-3.1.0-2.el9.noarch.rpm 179 kB/s | 16 kB 00:00 2023-12-15T22:53:40.609 INFO:teuthology.orchestra.run.smithi072.stdout:(29/35): python3-pyOpenSSL-21.0.0-1.el9.noarch. 898 kB/s | 90 kB 00:00 2023-12-15T22:53:40.701 INFO:teuthology.orchestra.run.smithi072.stdout:(30/35): python3-tempora-5.0.0-2.el9.noarch.rpm 390 kB/s | 36 kB 00:00 2023-12-15T22:53:40.818 INFO:teuthology.orchestra.run.smithi072.stdout:(31/35): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2023-12-15T22:53:40.960 INFO:teuthology.orchestra.run.smithi072.stdout:(32/35): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.9 MB/s | 427 kB 00:00 2023-12-15T22:53:41.052 INFO:teuthology.orchestra.run.smithi072.stdout:(33/35): python3-zc-lockfile-2.0-10.el9.noarch. 217 kB/s | 20 kB 00:00 2023-12-15T22:53:43.736 INFO:teuthology.orchestra.run.smithi072.stdout:(34/35): ceph-mon-19.0.0-201.gcd750a7f.el9.x86_ 870 kB/s | 8.9 MB 00:10 2023-12-15T22:54:10.853 INFO:teuthology.orchestra.run.smithi072.stdout:(35/35): ceph-osd-19.0.0-201.gcd750a7f.el9.x86_ 3.3 MB/s | 117 MB 00:35 2023-12-15T22:54:10.857 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2023-12-15T22:54:10.857 INFO:teuthology.orchestra.run.smithi072.stdout:Total 3.5 MB/s | 134 MB 00:38 2023-12-15T22:54:10.858 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2023-12-15T22:54:10.894 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2023-12-15T22:54:10.894 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2023-12-15T22:54:11.696 INFO:teuthology.orchestra.run.smithi071.stdout:(35/35): ceph-osd-19.0.0-201.gcd750a7f.el9.x86_ 1.8 MB/s | 117 MB 01:03 2023-12-15T22:54:11.699 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-12-15T22:54:11.700 INFO:teuthology.orchestra.run.smithi071.stdout:Total 2.1 MB/s | 134 MB 01:04 2023-12-15T22:54:11.700 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-12-15T22:54:11.737 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-12-15T22:54:11.737 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-12-15T22:54:11.930 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2023-12-15T22:54:11.930 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2023-12-15T22:54:12.761 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-12-15T22:54:12.761 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-12-15T22:54:13.830 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2023-12-15T22:54:13.898 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/35 2023-12-15T22:54:13.938 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/35 2023-12-15T22:54:13.980 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/35 2023-12-15T22:54:14.071 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/35 2023-12-15T22:54:14.159 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/35 2023-12-15T22:54:14.195 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/35 2023-12-15T22:54:14.275 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/35 2023-12-15T22:54:14.338 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 8/35 2023-12-15T22:54:14.377 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 9/35 2023-12-15T22:54:14.435 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/35 2023-12-15T22:54:14.502 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/35 2023-12-15T22:54:14.674 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-12-15T22:54:14.706 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/35 2023-12-15T22:54:14.740 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/35 2023-12-15T22:54:14.762 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/35 2023-12-15T22:54:14.780 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/35 2023-12-15T22:54:14.801 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/35 2023-12-15T22:54:14.813 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/35 2023-12-15T22:54:14.836 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/35 2023-12-15T22:54:14.867 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/35 2023-12-15T22:54:14.903 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/35 2023-12-15T22:54:14.953 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/35 2023-12-15T22:54:14.982 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/35 2023-12-15T22:54:14.989 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/35 2023-12-15T22:54:15.023 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/35 2023-12-15T22:54:15.032 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/35 2023-12-15T22:54:15.063 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/35 2023-12-15T22:54:15.099 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/35 2023-12-15T22:54:15.104 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/35 2023-12-15T22:54:15.126 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/35 2023-12-15T22:54:15.158 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 8/35 2023-12-15T22:54:15.196 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 9/35 2023-12-15T22:54:15.207 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/35 2023-12-15T22:54:15.245 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/35 2023-12-15T22:54:15.251 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/35 2023-12-15T22:54:15.304 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/35 2023-12-15T22:54:15.324 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/35 2023-12-15T22:54:15.486 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/35 2023-12-15T22:54:15.512 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/35 2023-12-15T22:54:15.542 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/35 2023-12-15T22:54:15.573 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/35 2023-12-15T22:54:15.607 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/35 2023-12-15T22:54:15.630 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/35 2023-12-15T22:54:15.637 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/35 2023-12-15T22:54:15.714 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/35 2023-12-15T22:54:15.748 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/35 2023-12-15T22:54:15.781 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/35 2023-12-15T22:54:15.799 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-201.gcd750a7f.el9.n 28/35 2023-12-15T22:54:15.821 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/35 2023-12-15T22:54:15.830 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-2:19.0.0-201.gcd750a7f.el9.x86_64 29/35 2023-12-15T22:54:15.855 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-2:19.0.0-201.gcd750a7f.el9.x86_64 29/35 2023-12-15T22:54:15.855 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-15T22:54:15.855 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-15T22:54:15.855 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-15T22:54:15.856 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-15T22:54:15.856 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:54:15.865 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/35 2023-12-15T22:54:15.899 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/35 2023-12-15T22:54:15.947 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : lua-5.4.4-4.el9.x86_64 30/35 2023-12-15T22:54:15.988 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/35 2023-12-15T22:54:16.022 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/35 2023-12-15T22:54:16.093 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/35 2023-12-15T22:54:16.238 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/35 2023-12-15T22:54:16.363 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/35 2023-12-15T22:54:16.525 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-201.gcd750a7f.el9.n 28/35 2023-12-15T22:54:16.552 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-2:19.0.0-201.gcd750a7f.el9.x86_64 29/35 2023-12-15T22:54:16.576 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-2:19.0.0-201.gcd750a7f.el9.x86_64 29/35 2023-12-15T22:54:16.576 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-15T22:54:16.577 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-15T22:54:16.577 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-15T22:54:16.577 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-15T22:54:16.577 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:54:16.650 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : lua-5.4.4-4.el9.x86_64 30/35 2023-12-15T22:54:34.301 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : luarocks-3.9.2-1.el9.noarch 31/35 2023-12-15T22:54:34.330 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-osd-2:19.0.0-201.gcd750a7f.el9.x86_64 32/35 2023-12-15T22:54:34.356 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-osd-2:19.0.0-201.gcd750a7f.el9.x86_64 32/35 2023-12-15T22:54:34.356 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-15T22:54:34.357 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-15T22:54:34.357 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-15T22:54:34.357 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-15T22:54:34.357 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:54:35.070 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : luarocks-3.9.2-1.el9.noarch 31/35 2023-12-15T22:54:35.239 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mon-2:19.0.0-201.gcd750a7f.el9.x86_64 33/35 2023-12-15T22:54:35.262 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mon-2:19.0.0-201.gcd750a7f.el9.x86_64 33/35 2023-12-15T22:54:35.262 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-15T22:54:35.262 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-15T22:54:35.262 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-15T22:54:35.262 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-15T22:54:35.262 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:54:35.361 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-osd-2:19.0.0-201.gcd750a7f.el9.x86_64 32/35 2023-12-15T22:54:35.388 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-osd-2:19.0.0-201.gcd750a7f.el9.x86_64 32/35 2023-12-15T22:54:35.388 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-15T22:54:35.388 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-15T22:54:35.388 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-15T22:54:35.388 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-15T22:54:35.388 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:54:35.474 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mds-2:19.0.0-201.gcd750a7f.el9.x86_64 34/35 2023-12-15T22:54:35.497 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mds-2:19.0.0-201.gcd750a7f.el9.x86_64 34/35 2023-12-15T22:54:35.497 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-15T22:54:35.497 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-15T22:54:35.497 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-15T22:54:35.497 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-15T22:54:35.497 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:54:35.563 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-2:19.0.0-201.gcd750a7f.el9.x86_64 35/35 2023-12-15T22:54:36.278 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mon-2:19.0.0-201.gcd750a7f.el9.x86_64 33/35 2023-12-15T22:54:36.301 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mon-2:19.0.0-201.gcd750a7f.el9.x86_64 33/35 2023-12-15T22:54:36.301 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-15T22:54:36.301 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-15T22:54:36.301 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-15T22:54:36.301 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-15T22:54:36.301 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:54:36.503 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mds-2:19.0.0-201.gcd750a7f.el9.x86_64 34/35 2023-12-15T22:54:36.526 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mds-2:19.0.0-201.gcd750a7f.el9.x86_64 34/35 2023-12-15T22:54:36.526 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-15T22:54:36.526 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-15T22:54:36.526 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-15T22:54:36.526 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-15T22:54:36.526 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:54:36.591 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-2:19.0.0-201.gcd750a7f.el9.x86_64 35/35 2023-12-15T22:54:46.665 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-2:19.0.0-201.gcd750a7f.el9.x86_64 35/35 2023-12-15T22:54:46.665 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-2:19.0.0-201.gcd750a7f.el9.x86_64 1/35 2023-12-15T22:54:46.665 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mds-2:19.0.0-201.gcd750a7f.el9.x86_64 2/35 2023-12-15T22:54:46.665 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-2:19.0.0-201.gcd750a7f.el9.x86_64 3/35 2023-12-15T22:54:46.665 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mon-2:19.0.0-201.gcd750a7f.el9.x86_64 4/35 2023-12-15T22:54:46.665 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-osd-2:19.0.0-201.gcd750a7f.el9.x86_64 5/35 2023-12-15T22:54:46.665 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-201.gcd750a7f.el9.n 6/35 2023-12-15T22:54:46.665 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/35 2023-12-15T22:54:46.665 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/35 2023-12-15T22:54:46.665 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/35 2023-12-15T22:54:46.665 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/35 2023-12-15T22:54:46.666 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/35 2023-12-15T22:54:46.666 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/35 2023-12-15T22:54:46.666 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/35 2023-12-15T22:54:46.666 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/35 2023-12-15T22:54:46.666 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/35 2023-12-15T22:54:46.666 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/35 2023-12-15T22:54:46.666 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 17/35 2023-12-15T22:54:46.666 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 18/35 2023-12-15T22:54:46.666 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 19/35 2023-12-15T22:54:46.666 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 20/35 2023-12-15T22:54:46.666 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 21/35 2023-12-15T22:54:46.666 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 22/35 2023-12-15T22:54:46.666 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 23/35 2023-12-15T22:54:46.666 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 24/35 2023-12-15T22:54:46.666 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 25/35 2023-12-15T22:54:46.667 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 26/35 2023-12-15T22:54:46.668 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 27/35 2023-12-15T22:54:46.668 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 28/35 2023-12-15T22:54:46.668 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 29/35 2023-12-15T22:54:46.668 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 30/35 2023-12-15T22:54:46.668 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 31/35 2023-12-15T22:54:46.668 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 32/35 2023-12-15T22:54:46.668 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 33/35 2023-12-15T22:54:46.668 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 34/35 2023-12-15T22:54:47.537 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 35/35 2023-12-15T22:54:47.537 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:54:47.537 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-12-15T22:54:47.537 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:54:47.537 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mds-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:54:47.537 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:54:47.537 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-modules-core-2:19.0.0-201.gcd750a7f.el9.noarch 2023-12-15T22:54:47.537 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mon-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:54:47.537 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-osd-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:54:47.538 INFO:teuthology.orchestra.run.smithi071.stdout: lua-5.4.4-4.el9.x86_64 2023-12-15T22:54:47.538 INFO:teuthology.orchestra.run.smithi071.stdout: luarocks-3.9.2-1.el9.noarch 2023-12-15T22:54:47.538 INFO:teuthology.orchestra.run.smithi071.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-15T22:54:47.538 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-15T22:54:47.538 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-15T22:54:47.538 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-15T22:54:47.538 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-15T22:54:47.538 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-15T22:54:47.538 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-15T22:54:47.538 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-15T22:54:47.538 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-15T22:54:47.538 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-15T22:54:47.538 INFO:teuthology.orchestra.run.smithi071.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-15T22:54:47.539 INFO:teuthology.orchestra.run.smithi071.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-15T22:54:47.539 INFO:teuthology.orchestra.run.smithi071.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-15T22:54:47.539 INFO:teuthology.orchestra.run.smithi071.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-15T22:54:47.539 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-15T22:54:47.539 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ply-3.11-14.el9.noarch 2023-12-15T22:54:47.539 INFO:teuthology.orchestra.run.smithi071.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-15T22:54:47.539 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-15T22:54:47.539 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-15T22:54:47.539 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-15T22:54:47.539 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-15T22:54:47.539 INFO:teuthology.orchestra.run.smithi071.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-15T22:54:47.539 INFO:teuthology.orchestra.run.smithi071.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-15T22:54:47.539 INFO:teuthology.orchestra.run.smithi071.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-15T22:54:47.539 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-15T22:54:47.540 INFO:teuthology.orchestra.run.smithi071.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-15T22:54:47.540 INFO:teuthology.orchestra.run.smithi071.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-15T22:54:47.540 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:54:47.540 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-12-15T22:54:47.791 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-base 2023-12-15T22:54:48.378 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:24 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:54:48.490 INFO:teuthology.orchestra.run.smithi071.stdout:Package ceph-base-2:19.0.0-201.gcd750a7f.el9.x86_64 is already installed. 2023-12-15T22:54:48.539 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-12-15T22:54:48.540 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2023-12-15T22:54:48.540 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-12-15T22:54:48.608 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install cephadm 2023-12-15T22:54:49.165 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:25 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:54:49.309 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-12-15T22:54:49.310 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-12-15T22:54:49.310 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-12-15T22:54:49.310 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-12-15T22:54:49.310 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-12-15T22:54:49.310 INFO:teuthology.orchestra.run.smithi071.stdout: cephadm noarch 2:19.0.0-201.gcd750a7f.el9 ceph-noarch 807 k 2023-12-15T22:54:49.310 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:54:49.310 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-12-15T22:54:49.310 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-12-15T22:54:49.310 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2023-12-15T22:54:49.310 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:54:49.310 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 807 k 2023-12-15T22:54:49.310 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 823 k 2023-12-15T22:54:49.311 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-12-15T22:54:50.192 INFO:teuthology.orchestra.run.smithi071.stdout:cephadm-19.0.0-201.gcd750a7f.el9.noarch.rpm 916 kB/s | 807 kB 00:00 2023-12-15T22:54:50.193 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-12-15T22:54:50.193 INFO:teuthology.orchestra.run.smithi071.stdout:Total 914 kB/s | 807 kB 00:00 2023-12-15T22:54:50.193 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-12-15T22:54:50.199 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-12-15T22:54:50.199 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-12-15T22:54:50.205 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-12-15T22:54:50.205 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-12-15T22:54:50.283 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-12-15T22:54:50.697 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: cephadm-2:19.0.0-201.gcd750a7f.el9.noarch 1/1 2023-12-15T22:54:50.789 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : cephadm-2:19.0.0-201.gcd750a7f.el9.noarch 1/1 2023-12-15T22:54:51.150 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: cephadm-2:19.0.0-201.gcd750a7f.el9.noarch 1/1 2023-12-15T22:54:51.445 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : cephadm-2:19.0.0-201.gcd750a7f.el9.noarch 1/1 2023-12-15T22:54:51.445 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:54:51.445 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-12-15T22:54:51.445 INFO:teuthology.orchestra.run.smithi071.stdout: cephadm-2:19.0.0-201.gcd750a7f.el9.noarch 2023-12-15T22:54:51.445 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:54:51.445 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-12-15T22:54:51.662 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-immutable-object-cache 2023-12-15T22:54:52.220 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:28 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:54:52.368 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-12-15T22:54:52.369 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-12-15T22:54:52.369 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2023-12-15T22:54:52.369 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-12-15T22:54:52.369 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-12-15T22:54:52.369 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 144 k 2023-12-15T22:54:52.369 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:54:52.369 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-12-15T22:54:52.369 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-12-15T22:54:52.369 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2023-12-15T22:54:52.369 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:54:52.369 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 144 k 2023-12-15T22:54:52.369 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 438 k 2023-12-15T22:54:52.370 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-12-15T22:54:53.072 INFO:teuthology.orchestra.run.smithi071.stdout:ceph-immutable-object-cache-19.0.0-201.gcd750a7 205 kB/s | 144 kB 00:00 2023-12-15T22:54:53.072 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-12-15T22:54:53.073 INFO:teuthology.orchestra.run.smithi071.stdout:Total 204 kB/s | 144 kB 00:00 2023-12-15T22:54:53.073 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-12-15T22:54:53.081 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-12-15T22:54:53.081 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-12-15T22:54:53.137 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-12-15T22:54:53.137 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-12-15T22:54:53.296 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-12-15T22:54:53.364 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-201.gcd750a7f.e 1/1 2023-12-15T22:54:53.388 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-201.gcd750a7f.e 1/1 2023-12-15T22:54:53.388 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-15T22:54:53.388 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-15T22:54:53.389 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:54:54.420 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-201.gcd750a7f.e 1/1 2023-12-15T22:54:54.420 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:54:54.420 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-12-15T22:54:54.420 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-immutable-object-cache-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:54:54.420 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:54:54.420 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-12-15T22:54:54.641 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr 2023-12-15T22:54:55.203 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:31 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:54:55.295 INFO:teuthology.orchestra.run.smithi071.stdout:Package ceph-mgr-2:19.0.0-201.gcd750a7f.el9.x86_64 is already installed. 2023-12-15T22:54:55.345 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-12-15T22:54:55.346 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2023-12-15T22:54:55.346 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-12-15T22:54:55.386 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-2:19.0.0-201.gcd750a7f.el9.x86_64 35/35 2023-12-15T22:54:55.386 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-2:19.0.0-201.gcd750a7f.el9.x86_64 1/35 2023-12-15T22:54:55.386 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mds-2:19.0.0-201.gcd750a7f.el9.x86_64 2/35 2023-12-15T22:54:55.386 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-2:19.0.0-201.gcd750a7f.el9.x86_64 3/35 2023-12-15T22:54:55.386 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mon-2:19.0.0-201.gcd750a7f.el9.x86_64 4/35 2023-12-15T22:54:55.386 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-osd-2:19.0.0-201.gcd750a7f.el9.x86_64 5/35 2023-12-15T22:54:55.386 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-201.gcd750a7f.el9.n 6/35 2023-12-15T22:54:55.387 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/35 2023-12-15T22:54:55.387 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/35 2023-12-15T22:54:55.387 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/35 2023-12-15T22:54:55.387 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/35 2023-12-15T22:54:55.387 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/35 2023-12-15T22:54:55.387 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/35 2023-12-15T22:54:55.387 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/35 2023-12-15T22:54:55.387 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/35 2023-12-15T22:54:55.387 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/35 2023-12-15T22:54:55.387 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/35 2023-12-15T22:54:55.387 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 17/35 2023-12-15T22:54:55.387 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 18/35 2023-12-15T22:54:55.387 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 19/35 2023-12-15T22:54:55.387 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 20/35 2023-12-15T22:54:55.387 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 21/35 2023-12-15T22:54:55.387 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 22/35 2023-12-15T22:54:55.388 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 23/35 2023-12-15T22:54:55.388 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 24/35 2023-12-15T22:54:55.388 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 25/35 2023-12-15T22:54:55.388 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 26/35 2023-12-15T22:54:55.388 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 27/35 2023-12-15T22:54:55.388 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 28/35 2023-12-15T22:54:55.388 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 29/35 2023-12-15T22:54:55.388 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 30/35 2023-12-15T22:54:55.388 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 31/35 2023-12-15T22:54:55.388 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 32/35 2023-12-15T22:54:55.388 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 33/35 2023-12-15T22:54:55.388 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 34/35 2023-12-15T22:54:55.413 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-dashboard 2023-12-15T22:54:55.962 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:31 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:54:56.108 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-12-15T22:54:56.109 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-12-15T22:54:56.109 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-12-15T22:54:56.109 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-12-15T22:54:56.109 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-12-15T22:54:56.109 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-dashboard noarch 2:19.0.0-201.gcd750a7f.el9 ceph-noarch 1.8 M 2023-12-15T22:54:56.109 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2023-12-15T22:54:56.109 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-grafana-dashboards noarch 2:19.0.0-201.gcd750a7f.el9 ceph-noarch 28 k 2023-12-15T22:54:56.109 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-prometheus-alerts noarch 2:19.0.0-201.gcd750a7f.el9 ceph-noarch 15 k 2023-12-15T22:54:56.109 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-15T22:54:56.109 INFO:teuthology.orchestra.run.smithi071.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-15T22:54:56.110 INFO:teuthology.orchestra.run.smithi071.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-15T22:54:56.110 INFO:teuthology.orchestra.run.smithi071.stdout:Installing weak dependencies: 2023-12-15T22:54:56.110 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-15T22:54:56.110 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:54:56.110 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-12-15T22:54:56.110 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-12-15T22:54:56.110 INFO:teuthology.orchestra.run.smithi071.stdout:Install 7 Packages 2023-12-15T22:54:56.110 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:54:56.110 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 2.1 M 2023-12-15T22:54:56.110 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 8.7 M 2023-12-15T22:54:56.111 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-12-15T22:54:56.349 INFO:teuthology.orchestra.run.smithi071.stdout:(1/7): ceph-grafana-dashboards-19.0.0-201.gcd75 149 kB/s | 28 kB 00:00 2023-12-15T22:54:56.374 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 35/35 2023-12-15T22:54:56.374 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:54:56.374 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2023-12-15T22:54:56.374 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:54:56.374 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mds-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:54:56.374 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:54:56.374 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-modules-core-2:19.0.0-201.gcd750a7f.el9.noarch 2023-12-15T22:54:56.374 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mon-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:54:56.374 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-osd-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:54:56.374 INFO:teuthology.orchestra.run.smithi072.stdout: lua-5.4.4-4.el9.x86_64 2023-12-15T22:54:56.375 INFO:teuthology.orchestra.run.smithi072.stdout: luarocks-3.9.2-1.el9.noarch 2023-12-15T22:54:56.375 INFO:teuthology.orchestra.run.smithi072.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-15T22:54:56.375 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-15T22:54:56.375 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-15T22:54:56.375 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-15T22:54:56.375 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-15T22:54:56.375 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-15T22:54:56.375 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-15T22:54:56.375 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-15T22:54:56.375 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-15T22:54:56.375 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-15T22:54:56.375 INFO:teuthology.orchestra.run.smithi072.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-15T22:54:56.375 INFO:teuthology.orchestra.run.smithi072.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-15T22:54:56.375 INFO:teuthology.orchestra.run.smithi072.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-15T22:54:56.375 INFO:teuthology.orchestra.run.smithi072.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-15T22:54:56.375 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-15T22:54:56.375 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ply-3.11-14.el9.noarch 2023-12-15T22:54:56.376 INFO:teuthology.orchestra.run.smithi072.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-15T22:54:56.376 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-15T22:54:56.376 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-15T22:54:56.376 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-15T22:54:56.376 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-15T22:54:56.376 INFO:teuthology.orchestra.run.smithi072.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-15T22:54:56.376 INFO:teuthology.orchestra.run.smithi072.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-15T22:54:56.376 INFO:teuthology.orchestra.run.smithi072.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-15T22:54:56.376 INFO:teuthology.orchestra.run.smithi072.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-15T22:54:56.376 INFO:teuthology.orchestra.run.smithi072.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-15T22:54:56.376 INFO:teuthology.orchestra.run.smithi072.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-15T22:54:56.376 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:54:56.376 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-12-15T22:54:56.542 INFO:teuthology.orchestra.run.smithi071.stdout:(2/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 47 kB/s | 9.0 kB 00:00 2023-12-15T22:54:56.567 INFO:teuthology.orchestra.run.smithi071.stdout:(3/7): ceph-prometheus-alerts-19.0.0-201.gcd750 37 kB/s | 15 kB 00:00 2023-12-15T22:54:56.675 INFO:teuthology.orchestra.run.smithi071.stdout:(4/7): ceph-mgr-dashboard-19.0.0-201.gcd750a7f. 3.4 MB/s | 1.8 MB 00:00 2023-12-15T22:54:56.679 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-base 2023-12-15T22:54:56.759 INFO:teuthology.orchestra.run.smithi071.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 188 kB/s | 41 kB 00:00 2023-12-15T22:54:56.884 INFO:teuthology.orchestra.run.smithi071.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 97 kB/s | 31 kB 00:00 2023-12-15T22:54:56.942 INFO:teuthology.orchestra.run.smithi071.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 706 kB/s | 188 kB 00:00 2023-12-15T22:54:56.944 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-12-15T22:54:56.944 INFO:teuthology.orchestra.run.smithi071.stdout:Total 2.5 MB/s | 2.1 MB 00:00 2023-12-15T22:54:56.945 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-12-15T22:54:56.953 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-12-15T22:54:56.953 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-12-15T22:54:57.017 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-12-15T22:54:57.017 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-12-15T22:54:57.247 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:04:33 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:54:57.313 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-12-15T22:54:57.341 INFO:teuthology.orchestra.run.smithi072.stdout:Package ceph-base-2:19.0.0-201.gcd750a7f.el9.x86_64 is already installed. 2023-12-15T22:54:57.377 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-12-15T22:54:57.392 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-12-15T22:54:57.393 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2023-12-15T22:54:57.393 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-12-15T22:54:57.411 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-12-15T22:54:57.466 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-12-15T22:54:57.467 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install cephadm 2023-12-15T22:54:57.506 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-12-15T22:54:57.536 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-201.gcd750a7f.el9.no 5/7 2023-12-15T22:54:57.774 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-201.gcd750a7f.el9.n 6/7 2023-12-15T22:54:57.804 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-201.gcd750a7f.el9.noarch 7/7 2023-12-15T22:54:58.038 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:04:34 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:54:58.180 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-12-15T22:54:58.181 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-12-15T22:54:58.181 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2023-12-15T22:54:58.181 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-12-15T22:54:58.181 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2023-12-15T22:54:58.181 INFO:teuthology.orchestra.run.smithi072.stdout: cephadm noarch 2:19.0.0-201.gcd750a7f.el9 ceph-noarch 807 k 2023-12-15T22:54:58.181 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:54:58.181 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2023-12-15T22:54:58.181 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-12-15T22:54:58.181 INFO:teuthology.orchestra.run.smithi072.stdout:Install 1 Package 2023-12-15T22:54:58.182 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:54:58.182 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 807 k 2023-12-15T22:54:58.182 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 823 k 2023-12-15T22:54:58.182 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2023-12-15T22:54:58.336 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-201.gcd750a7f.el9.noarch 7/7 2023-12-15T22:54:58.336 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-201.gcd750a7f.el9.n 1/7 2023-12-15T22:54:58.336 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-201.gcd750a7f.el9.noarch 2/7 2023-12-15T22:54:58.336 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-201.gcd750a7f.el9.no 3/7 2023-12-15T22:54:58.336 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-12-15T22:54:58.336 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-12-15T22:54:58.336 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-12-15T22:54:58.651 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-12-15T22:54:58.651 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:54:58.651 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-12-15T22:54:58.651 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-grafana-dashboards-2:19.0.0-201.gcd750a7f.el9.noarch 2023-12-15T22:54:58.651 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-dashboard-2:19.0.0-201.gcd750a7f.el9.noarch 2023-12-15T22:54:58.651 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-prometheus-alerts-2:19.0.0-201.gcd750a7f.el9.noarch 2023-12-15T22:54:58.651 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-15T22:54:58.651 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-15T22:54:58.651 INFO:teuthology.orchestra.run.smithi071.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-15T22:54:58.651 INFO:teuthology.orchestra.run.smithi071.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-15T22:54:58.651 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:54:58.651 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-12-15T22:54:58.893 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-15T22:54:58.951 INFO:teuthology.orchestra.run.smithi072.stdout:cephadm-19.0.0-201.gcd750a7f.el9.noarch.rpm 1.0 MB/s | 807 kB 00:00 2023-12-15T22:54:58.951 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2023-12-15T22:54:58.952 INFO:teuthology.orchestra.run.smithi072.stdout:Total 1.0 MB/s | 807 kB 00:00 2023-12-15T22:54:58.952 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2023-12-15T22:54:58.957 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2023-12-15T22:54:58.957 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2023-12-15T22:54:58.964 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2023-12-15T22:54:58.964 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2023-12-15T22:54:59.037 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2023-12-15T22:54:59.448 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:35 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:54:59.501 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: cephadm-2:19.0.0-201.gcd750a7f.el9.noarch 1/1 2023-12-15T22:54:59.592 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-12-15T22:54:59.594 INFO:teuthology.orchestra.run.smithi071.stdout:=========================================================================================== 2023-12-15T22:54:59.594 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-12-15T22:54:59.594 INFO:teuthology.orchestra.run.smithi071.stdout:=========================================================================================== 2023-12-15T22:54:59.594 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-12-15T22:54:59.594 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-201.gcd750a7f.el9 ceph-noarch 7.4 M 2023-12-15T22:54:59.594 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2023-12-15T22:54:59.594 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-15T22:54:59.594 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-15T22:54:59.594 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-15T22:54:59.594 INFO:teuthology.orchestra.run.smithi071.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-15T22:54:59.594 INFO:teuthology.orchestra.run.smithi071.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-15T22:54:59.594 INFO:teuthology.orchestra.run.smithi071.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-15T22:54:59.595 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-15T22:54:59.595 INFO:teuthology.orchestra.run.smithi071.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-15T22:54:59.595 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-15T22:54:59.595 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-15T22:54:59.595 INFO:teuthology.orchestra.run.smithi071.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-15T22:54:59.595 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:54:59.595 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-12-15T22:54:59.595 INFO:teuthology.orchestra.run.smithi071.stdout:=========================================================================================== 2023-12-15T22:54:59.595 INFO:teuthology.orchestra.run.smithi071.stdout:Install 12 Packages 2023-12-15T22:54:59.595 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:54:59.596 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : cephadm-2:19.0.0-201.gcd750a7f.el9.noarch 1/1 2023-12-15T22:54:59.596 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 37 M 2023-12-15T22:54:59.596 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 209 M 2023-12-15T22:54:59.596 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-12-15T22:54:59.999 INFO:teuthology.orchestra.run.smithi071.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 243 kB/s | 32 kB 00:00 2023-12-15T22:55:00.002 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: cephadm-2:19.0.0-201.gcd750a7f.el9.noarch 1/1 2023-12-15T22:55:00.041 INFO:teuthology.orchestra.run.smithi071.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 423 kB/s | 17 kB 00:00 2023-12-15T22:55:00.075 INFO:teuthology.orchestra.run.smithi071.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2023-12-15T22:55:00.167 INFO:teuthology.orchestra.run.smithi071.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2023-12-15T22:55:00.242 INFO:teuthology.orchestra.run.smithi071.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 28 MB/s | 4.7 MB 00:00 2023-12-15T22:55:00.307 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : cephadm-2:19.0.0-201.gcd750a7f.el9.noarch 1/1 2023-12-15T22:55:00.307 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:00.307 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2023-12-15T22:55:00.307 INFO:teuthology.orchestra.run.smithi072.stdout: cephadm-2:19.0.0-201.gcd750a7f.el9.noarch 2023-12-15T22:55:00.307 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:00.307 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-12-15T22:55:00.317 INFO:teuthology.orchestra.run.smithi071.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2023-12-15T22:55:00.342 INFO:teuthology.orchestra.run.smithi071.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2023-12-15T22:55:00.544 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-immutable-object-cache 2023-12-15T22:55:00.584 INFO:teuthology.orchestra.run.smithi071.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 59 MB/s | 16 MB 00:00 2023-12-15T22:55:00.634 INFO:teuthology.orchestra.run.smithi071.stdout:(9/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 2.7 MB/s | 803 kB 00:00 2023-12-15T22:55:00.660 INFO:teuthology.orchestra.run.smithi071.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 2.6 MB/s | 195 kB 00:00 2023-12-15T22:55:00.752 INFO:teuthology.orchestra.run.smithi071.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 2.1 MB/s | 245 kB 00:00 2023-12-15T22:55:01.104 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:04:37 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:55:01.250 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-12-15T22:55:01.251 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-12-15T22:55:01.251 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repo Size 2023-12-15T22:55:01.251 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-12-15T22:55:01.251 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2023-12-15T22:55:01.251 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 144 k 2023-12-15T22:55:01.251 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:01.252 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2023-12-15T22:55:01.252 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-12-15T22:55:01.252 INFO:teuthology.orchestra.run.smithi072.stdout:Install 1 Package 2023-12-15T22:55:01.252 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:01.252 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 144 k 2023-12-15T22:55:01.252 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 438 k 2023-12-15T22:55:01.252 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2023-12-15T22:55:01.302 INFO:teuthology.orchestra.run.smithi071.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-2 5.2 MB/s | 7.4 MB 00:01 2023-12-15T22:55:01.304 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-12-15T22:55:01.305 INFO:teuthology.orchestra.run.smithi071.stdout:Total 22 MB/s | 37 MB 00:01 2023-12-15T22:55:01.318 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-12-15T22:55:01.342 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-12-15T22:55:01.342 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-12-15T22:55:01.599 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-immutable-object-cache-19.0.0-201.gcd750a7 413 kB/s | 144 kB 00:00 2023-12-15T22:55:01.600 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2023-12-15T22:55:01.600 INFO:teuthology.orchestra.run.smithi072.stdout:Total 411 kB/s | 144 kB 00:00 2023-12-15T22:55:01.600 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2023-12-15T22:55:01.607 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2023-12-15T22:55:01.608 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2023-12-15T22:55:01.660 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2023-12-15T22:55:01.660 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2023-12-15T22:55:01.734 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-12-15T22:55:01.734 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-12-15T22:55:01.809 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2023-12-15T22:55:01.886 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-201.gcd750a7f.e 1/1 2023-12-15T22:55:01.910 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-201.gcd750a7f.e 1/1 2023-12-15T22:55:01.910 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-15T22:55:01.910 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-15T22:55:01.910 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:02.437 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-12-15T22:55:02.508 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-12-15T22:55:02.547 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-12-15T22:55:02.640 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-12-15T22:55:02.676 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-12-15T22:55:02.853 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-15T22:55:02.883 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-15T22:55:02.964 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-12-15T22:55:02.967 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-201.gcd750a7f.e 1/1 2023-12-15T22:55:02.967 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:02.967 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2023-12-15T22:55:02.968 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-immutable-object-cache-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:55:02.968 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:02.968 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-12-15T22:55:03.221 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-mgr 2023-12-15T22:55:03.608 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-12-15T22:55:03.721 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-12-15T22:55:03.774 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:04:39 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:55:03.869 INFO:teuthology.orchestra.run.smithi072.stdout:Package ceph-mgr-2:19.0.0-201.gcd750a7f.el9.x86_64 is already installed. 2023-12-15T22:55:03.919 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-12-15T22:55:03.920 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2023-12-15T22:55:03.920 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-12-15T22:55:03.988 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-mgr-dashboard 2023-12-15T22:55:04.539 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:04:40 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:55:04.683 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-12-15T22:55:04.684 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-12-15T22:55:04.684 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2023-12-15T22:55:04.684 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-12-15T22:55:04.684 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2023-12-15T22:55:04.684 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-dashboard noarch 2:19.0.0-201.gcd750a7f.el9 ceph-noarch 1.8 M 2023-12-15T22:55:04.684 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2023-12-15T22:55:04.684 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-grafana-dashboards noarch 2:19.0.0-201.gcd750a7f.el9 ceph-noarch 28 k 2023-12-15T22:55:04.684 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-prometheus-alerts noarch 2:19.0.0-201.gcd750a7f.el9 ceph-noarch 15 k 2023-12-15T22:55:04.684 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-15T22:55:04.685 INFO:teuthology.orchestra.run.smithi072.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-15T22:55:04.685 INFO:teuthology.orchestra.run.smithi072.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-15T22:55:04.685 INFO:teuthology.orchestra.run.smithi072.stdout:Installing weak dependencies: 2023-12-15T22:55:04.685 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-15T22:55:04.685 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:04.685 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2023-12-15T22:55:04.685 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-12-15T22:55:04.685 INFO:teuthology.orchestra.run.smithi072.stdout:Install 7 Packages 2023-12-15T22:55:04.685 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:04.686 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 2.1 M 2023-12-15T22:55:04.686 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 8.7 M 2023-12-15T22:55:04.686 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2023-12-15T22:55:04.942 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-12-15T22:55:05.020 INFO:teuthology.orchestra.run.smithi072.stdout:(1/7): ceph-grafana-dashboards-19.0.0-201.gcd75 111 kB/s | 28 kB 00:00 2023-12-15T22:55:05.053 INFO:teuthology.orchestra.run.smithi072.stdout:(2/7): ceph-prometheus-alerts-19.0.0-201.gcd750 53 kB/s | 15 kB 00:00 2023-12-15T22:55:05.112 INFO:teuthology.orchestra.run.smithi072.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 98 kB/s | 9.0 kB 00:00 2023-12-15T22:55:05.221 INFO:teuthology.orchestra.run.smithi072.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 283 kB/s | 31 kB 00:00 2023-12-15T22:55:05.256 INFO:teuthology.orchestra.run.smithi072.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 202 kB/s | 41 kB 00:00 2023-12-15T22:55:05.381 INFO:teuthology.orchestra.run.smithi072.stdout:(6/7): ceph-mgr-dashboard-19.0.0-201.gcd750a7f. 2.9 MB/s | 1.8 MB 00:00 2023-12-15T22:55:05.407 INFO:teuthology.orchestra.run.smithi072.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.0 MB/s | 188 kB 00:00 2023-12-15T22:55:05.408 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2023-12-15T22:55:05.409 INFO:teuthology.orchestra.run.smithi072.stdout:Total 2.9 MB/s | 2.1 MB 00:00 2023-12-15T22:55:05.409 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2023-12-15T22:55:05.416 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2023-12-15T22:55:05.416 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2023-12-15T22:55:05.478 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2023-12-15T22:55:05.478 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2023-12-15T22:55:05.782 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2023-12-15T22:55:05.851 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-12-15T22:55:05.885 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-12-15T22:55:05.940 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-12-15T22:55:05.970 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-12-15T22:55:05.973 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-12-15T22:55:05.998 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-201.gcd750a 12/12 2023-12-15T22:55:06.012 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-201.gcd750a7f.el9.no 5/7 2023-12-15T22:55:06.250 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-201.gcd750a7f.el9.n 6/7 2023-12-15T22:55:06.291 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-201.gcd750a7f.el9.noarch 7/7 2023-12-15T22:55:06.967 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-201.gcd750a7f.el9.noarch 7/7 2023-12-15T22:55:06.967 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-201.gcd750a7f.el9.n 1/7 2023-12-15T22:55:06.967 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-201.gcd750a7f.el9.noarch 2/7 2023-12-15T22:55:06.967 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-201.gcd750a7f.el9.no 3/7 2023-12-15T22:55:06.967 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-12-15T22:55:06.967 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-12-15T22:55:06.967 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-12-15T22:55:07.288 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-12-15T22:55:07.288 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:07.288 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2023-12-15T22:55:07.288 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-grafana-dashboards-2:19.0.0-201.gcd750a7f.el9.noarch 2023-12-15T22:55:07.288 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-dashboard-2:19.0.0-201.gcd750a7f.el9.noarch 2023-12-15T22:55:07.288 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-prometheus-alerts-2:19.0.0-201.gcd750a7f.el9.noarch 2023-12-15T22:55:07.288 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-15T22:55:07.289 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-15T22:55:07.289 INFO:teuthology.orchestra.run.smithi072.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-15T22:55:07.289 INFO:teuthology.orchestra.run.smithi072.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-15T22:55:07.289 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:07.289 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-12-15T22:55:07.562 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-15T22:55:08.125 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:04:44 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:55:08.281 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-12-15T22:55:08.282 INFO:teuthology.orchestra.run.smithi072.stdout:=========================================================================================== 2023-12-15T22:55:08.282 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2023-12-15T22:55:08.283 INFO:teuthology.orchestra.run.smithi072.stdout:=========================================================================================== 2023-12-15T22:55:08.283 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2023-12-15T22:55:08.283 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-201.gcd750a7f.el9 ceph-noarch 7.4 M 2023-12-15T22:55:08.283 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2023-12-15T22:55:08.283 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-15T22:55:08.283 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-15T22:55:08.283 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-15T22:55:08.283 INFO:teuthology.orchestra.run.smithi072.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-15T22:55:08.283 INFO:teuthology.orchestra.run.smithi072.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-15T22:55:08.283 INFO:teuthology.orchestra.run.smithi072.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-15T22:55:08.283 INFO:teuthology.orchestra.run.smithi072.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-15T22:55:08.283 INFO:teuthology.orchestra.run.smithi072.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-15T22:55:08.283 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-15T22:55:08.284 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-15T22:55:08.284 INFO:teuthology.orchestra.run.smithi072.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-15T22:55:08.284 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:08.284 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2023-12-15T22:55:08.284 INFO:teuthology.orchestra.run.smithi072.stdout:=========================================================================================== 2023-12-15T22:55:08.284 INFO:teuthology.orchestra.run.smithi072.stdout:Install 12 Packages 2023-12-15T22:55:08.284 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:08.284 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 37 M 2023-12-15T22:55:08.285 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 209 M 2023-12-15T22:55:08.285 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2023-12-15T22:55:08.481 INFO:teuthology.orchestra.run.smithi072.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 238 kB/s | 32 kB 00:00 2023-12-15T22:55:08.514 INFO:teuthology.orchestra.run.smithi072.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 535 kB/s | 17 kB 00:00 2023-12-15T22:55:08.590 INFO:teuthology.orchestra.run.smithi072.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2023-12-15T22:55:08.617 INFO:teuthology.orchestra.run.smithi072.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 341 kB/s | 34 kB 00:00 2023-12-15T22:55:08.726 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-201.gcd750a 12/12 2023-12-15T22:55:08.726 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-201.gcd750a 1/12 2023-12-15T22:55:08.726 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-12-15T22:55:08.726 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-12-15T22:55:08.726 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-12-15T22:55:08.726 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-15T22:55:08.726 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-15T22:55:08.726 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-12-15T22:55:08.726 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-12-15T22:55:08.726 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-12-15T22:55:08.727 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-12-15T22:55:08.727 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-12-15T22:55:08.768 INFO:teuthology.orchestra.run.smithi072.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 27 MB/s | 4.7 MB 00:00 2023-12-15T22:55:08.868 INFO:teuthology.orchestra.run.smithi072.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2023-12-15T22:55:08.902 INFO:teuthology.orchestra.run.smithi072.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.6 MB/s | 217 kB 00:00 2023-12-15T22:55:08.978 INFO:teuthology.orchestra.run.smithi072.stdout:(8/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 11 MB/s | 803 kB 00:00 2023-12-15T22:55:09.172 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-12-15T22:55:09.173 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:09.173 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-12-15T22:55:09.173 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-diskprediction-local-2:19.0.0-201.gcd750a7f.el9.noarch 2023-12-15T22:55:09.173 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-15T22:55:09.173 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-15T22:55:09.173 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-15T22:55:09.173 INFO:teuthology.orchestra.run.smithi071.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-15T22:55:09.173 INFO:teuthology.orchestra.run.smithi071.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-15T22:55:09.173 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-15T22:55:09.173 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-15T22:55:09.173 INFO:teuthology.orchestra.run.smithi071.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-15T22:55:09.174 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-15T22:55:09.174 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-15T22:55:09.174 INFO:teuthology.orchestra.run.smithi071.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-15T22:55:09.174 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:09.174 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-12-15T22:55:09.245 INFO:teuthology.orchestra.run.smithi072.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 42 MB/s | 16 MB 00:00 2023-12-15T22:55:09.270 INFO:teuthology.orchestra.run.smithi072.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 672 kB/s | 195 kB 00:00 2023-12-15T22:55:09.571 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-rook 2023-12-15T22:55:09.587 INFO:teuthology.orchestra.run.smithi072.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-2 6.0 MB/s | 7.4 MB 00:01 2023-12-15T22:55:09.621 INFO:teuthology.orchestra.run.smithi072.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 654 kB/s | 245 kB 00:00 2023-12-15T22:55:09.624 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2023-12-15T22:55:09.624 INFO:teuthology.orchestra.run.smithi072.stdout:Total 28 MB/s | 37 MB 00:01 2023-12-15T22:55:09.637 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2023-12-15T22:55:09.662 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2023-12-15T22:55:09.662 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2023-12-15T22:55:10.041 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2023-12-15T22:55:10.041 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2023-12-15T22:55:10.132 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:46 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:55:10.276 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-12-15T22:55:10.277 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================= 2023-12-15T22:55:10.278 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-12-15T22:55:10.278 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================= 2023-12-15T22:55:10.278 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-12-15T22:55:10.278 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-rook noarch 2:19.0.0-201.gcd750a7f.el9 ceph-noarch 50 k 2023-12-15T22:55:10.278 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2023-12-15T22:55:10.278 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-15T22:55:10.278 INFO:teuthology.orchestra.run.smithi071.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-15T22:55:10.278 INFO:teuthology.orchestra.run.smithi071.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-15T22:55:10.278 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-15T22:55:10.278 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-15T22:55:10.278 INFO:teuthology.orchestra.run.smithi071.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-15T22:55:10.278 INFO:teuthology.orchestra.run.smithi071.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-15T22:55:10.278 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-15T22:55:10.279 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-15T22:55:10.279 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-15T22:55:10.279 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-15T22:55:10.279 INFO:teuthology.orchestra.run.smithi071.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-15T22:55:10.279 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:10.279 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-12-15T22:55:10.279 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================= 2023-12-15T22:55:10.279 INFO:teuthology.orchestra.run.smithi071.stdout:Install 13 Packages 2023-12-15T22:55:10.279 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:10.280 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 2.0 M 2023-12-15T22:55:10.280 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 26 M 2023-12-15T22:55:10.280 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-12-15T22:55:10.439 INFO:teuthology.orchestra.run.smithi071.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 169 kB/s | 18 kB 00:00 2023-12-15T22:55:10.455 INFO:teuthology.orchestra.run.smithi071.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 197 kB/s | 24 kB 00:00 2023-12-15T22:55:10.497 INFO:teuthology.orchestra.run.smithi071.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.9 MB/s | 174 kB 00:00 2023-12-15T22:55:10.523 INFO:teuthology.orchestra.run.smithi071.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2023-12-15T22:55:10.558 INFO:teuthology.orchestra.run.smithi071.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.4 MB/s | 211 kB 00:00 2023-12-15T22:55:10.575 INFO:teuthology.orchestra.run.smithi071.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 821 kB/s | 43 kB 00:00 2023-12-15T22:55:10.592 INFO:teuthology.orchestra.run.smithi071.stdout:(7/13): ceph-mgr-rook-19.0.0-201.gcd750a7f.el9. 192 kB/s | 50 kB 00:00 2023-12-15T22:55:10.718 INFO:teuthology.orchestra.run.smithi071.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 105 kB/s | 15 kB 00:00 2023-12-15T22:55:10.737 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2023-12-15T22:55:10.809 INFO:teuthology.orchestra.run.smithi071.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 128 kB/s | 32 kB 00:00 2023-12-15T22:55:10.819 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-12-15T22:55:10.851 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-12-15T22:55:10.926 INFO:teuthology.orchestra.run.smithi071.stdout:(10/13): python3-google-auth-2.23.4-1.el9.noarc 592 kB/s | 197 kB 00:00 2023-12-15T22:55:10.944 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-12-15T22:55:10.968 INFO:teuthology.orchestra.run.smithi071.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 373 kB/s | 59 kB 00:00 2023-12-15T22:55:10.983 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-12-15T22:55:11.160 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-15T22:55:11.168 INFO:teuthology.orchestra.run.smithi071.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 371 kB/s | 90 kB 00:00 2023-12-15T22:55:11.224 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-15T22:55:11.309 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-12-15T22:55:11.503 INFO:teuthology.orchestra.run.smithi071.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.3 MB/s | 1.0 MB 00:00 2023-12-15T22:55:11.505 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-12-15T22:55:11.505 INFO:teuthology.orchestra.run.smithi071.stdout:Total 1.7 MB/s | 2.0 MB 00:01 2023-12-15T22:55:11.506 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-12-15T22:55:11.519 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-12-15T22:55:11.519 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-12-15T22:55:11.617 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-12-15T22:55:11.618 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-12-15T22:55:11.956 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-12-15T22:55:12.076 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-12-15T22:55:12.133 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-12-15T22:55:12.315 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-12-15T22:55:12.373 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-12-15T22:55:12.450 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-12-15T22:55:12.487 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-12-15T22:55:12.537 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-12-15T22:55:12.637 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-12-15T22:55:12.739 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-12-15T22:55:12.789 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-12-15T22:55:13.303 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-12-15T22:55:13.504 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-12-15T22:55:13.560 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-12-15T22:55:13.594 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-12-15T22:55:13.641 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-12-15T22:55:13.664 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-rook-2:19.0.0-201.gcd750a7f.el9.noarch 13/13 2023-12-15T22:55:14.335 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-12-15T22:55:14.364 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-201.gcd750a 12/12 2023-12-15T22:55:14.451 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-201.gcd750a7f.el9.noarch 13/13 2023-12-15T22:55:14.451 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-rook-2:19.0.0-201.gcd750a7f.el9.noarch 1/13 2023-12-15T22:55:14.451 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-12-15T22:55:14.451 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-12-15T22:55:14.452 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-12-15T22:55:14.452 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-12-15T22:55:14.452 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-12-15T22:55:14.452 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-12-15T22:55:14.452 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-12-15T22:55:14.452 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-12-15T22:55:14.452 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-12-15T22:55:14.452 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-12-15T22:55:14.452 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-12-15T22:55:14.817 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-12-15T22:55:14.817 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:14.817 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-12-15T22:55:14.817 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-rook-2:19.0.0-201.gcd750a7f.el9.noarch 2023-12-15T22:55:14.817 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-15T22:55:14.817 INFO:teuthology.orchestra.run.smithi071.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-15T22:55:14.817 INFO:teuthology.orchestra.run.smithi071.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-15T22:55:14.817 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-15T22:55:14.817 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-15T22:55:14.817 INFO:teuthology.orchestra.run.smithi071.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-15T22:55:14.818 INFO:teuthology.orchestra.run.smithi071.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-15T22:55:14.818 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-15T22:55:14.818 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-15T22:55:14.818 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-15T22:55:14.818 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-15T22:55:14.818 INFO:teuthology.orchestra.run.smithi071.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-15T22:55:14.818 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:14.818 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-12-15T22:55:15.067 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-cephadm 2023-12-15T22:55:15.624 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:51 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:55:15.767 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-12-15T22:55:15.768 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================= 2023-12-15T22:55:15.768 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-12-15T22:55:15.768 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================= 2023-12-15T22:55:15.768 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-12-15T22:55:15.768 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-cephadm noarch 2:19.0.0-201.gcd750a7f.el9 ceph-noarch 133 k 2023-12-15T22:55:15.768 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2023-12-15T22:55:15.768 INFO:teuthology.orchestra.run.smithi071.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-15T22:55:15.768 INFO:teuthology.orchestra.run.smithi071.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-15T22:55:15.768 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-15T22:55:15.768 INFO:teuthology.orchestra.run.smithi071.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-15T22:55:15.769 INFO:teuthology.orchestra.run.smithi071.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-15T22:55:15.769 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:15.769 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-12-15T22:55:15.769 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================= 2023-12-15T22:55:15.769 INFO:teuthology.orchestra.run.smithi071.stdout:Install 6 Packages 2023-12-15T22:55:15.769 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:15.769 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 6.8 M 2023-12-15T22:55:15.769 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 33 M 2023-12-15T22:55:15.770 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-12-15T22:55:16.037 INFO:teuthology.orchestra.run.smithi071.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.5 MB/s | 228 kB 00:00 2023-12-15T22:55:16.187 INFO:teuthology.orchestra.run.smithi071.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2023-12-15T22:55:16.212 INFO:teuthology.orchestra.run.smithi071.stdout:(3/6): ceph-mgr-cephadm-19.0.0-201.gcd750a7f.el 410 kB/s | 133 kB 00:00 2023-12-15T22:55:16.338 INFO:teuthology.orchestra.run.smithi071.stdout:(4/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.8 MB/s | 548 kB 00:00 2023-12-15T22:55:16.363 INFO:teuthology.orchestra.run.smithi071.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 331 kB/s | 58 kB 00:00 2023-12-15T22:55:16.388 INFO:teuthology.orchestra.run.smithi071.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 291 kB/s | 51 kB 00:00 2023-12-15T22:55:16.391 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-12-15T22:55:16.392 INFO:teuthology.orchestra.run.smithi071.stdout:Total 11 MB/s | 6.8 MB 00:00 2023-12-15T22:55:16.392 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-12-15T22:55:16.401 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-12-15T22:55:16.401 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-12-15T22:55:16.489 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-12-15T22:55:16.489 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-12-15T22:55:16.791 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-12-15T22:55:16.968 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-12-15T22:55:17.022 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-12-15T22:55:17.540 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-12-15T22:55:17.616 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-12-15T22:55:17.677 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-12-15T22:55:17.702 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-201.gcd750a7f.el9.noarch 6/6 2023-12-15T22:55:18.300 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-201.gcd750a 12/12 2023-12-15T22:55:18.300 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-201.gcd750a 1/12 2023-12-15T22:55:18.301 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-12-15T22:55:18.301 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-12-15T22:55:18.301 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-12-15T22:55:18.301 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-15T22:55:18.301 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-15T22:55:18.301 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-12-15T22:55:18.301 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-12-15T22:55:18.301 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-12-15T22:55:18.301 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-12-15T22:55:18.301 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-12-15T22:55:18.543 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-201.gcd750a7f.el9.noarch 6/6 2023-12-15T22:55:18.543 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-201.gcd750a7f.el9.noarch 1/6 2023-12-15T22:55:18.543 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-12-15T22:55:18.543 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-12-15T22:55:18.543 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-12-15T22:55:18.543 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-12-15T22:55:18.737 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-12-15T22:55:18.737 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:18.737 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2023-12-15T22:55:18.737 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-diskprediction-local-2:19.0.0-201.gcd750a7f.el9.noarch 2023-12-15T22:55:18.737 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-15T22:55:18.737 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-15T22:55:18.737 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-15T22:55:18.737 INFO:teuthology.orchestra.run.smithi072.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-15T22:55:18.737 INFO:teuthology.orchestra.run.smithi072.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-15T22:55:18.737 INFO:teuthology.orchestra.run.smithi072.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-15T22:55:18.737 INFO:teuthology.orchestra.run.smithi072.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-15T22:55:18.737 INFO:teuthology.orchestra.run.smithi072.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-15T22:55:18.738 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-15T22:55:18.738 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-15T22:55:18.738 INFO:teuthology.orchestra.run.smithi072.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-15T22:55:18.738 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:18.738 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-12-15T22:55:18.866 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-12-15T22:55:18.866 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:18.866 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-12-15T22:55:18.866 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-cephadm-2:19.0.0-201.gcd750a7f.el9.noarch 2023-12-15T22:55:18.866 INFO:teuthology.orchestra.run.smithi071.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-15T22:55:18.866 INFO:teuthology.orchestra.run.smithi071.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-15T22:55:18.866 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-15T22:55:18.866 INFO:teuthology.orchestra.run.smithi071.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-15T22:55:18.866 INFO:teuthology.orchestra.run.smithi071.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-15T22:55:18.867 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:18.867 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-12-15T22:55:19.124 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-fuse 2023-12-15T22:55:19.137 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-mgr-rook 2023-12-15T22:55:19.683 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:55 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:55:19.697 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:04:55 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:55:19.827 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-12-15T22:55:19.828 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-12-15T22:55:19.828 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-12-15T22:55:19.828 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-12-15T22:55:19.828 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-12-15T22:55:19.828 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-fuse x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 832 k 2023-12-15T22:55:19.828 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2023-12-15T22:55:19.828 INFO:teuthology.orchestra.run.smithi071.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-15T22:55:19.828 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:19.829 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-12-15T22:55:19.829 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-12-15T22:55:19.829 INFO:teuthology.orchestra.run.smithi071.stdout:Install 2 Packages 2023-12-15T22:55:19.829 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:19.829 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 911 k 2023-12-15T22:55:19.830 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 3.0 M 2023-12-15T22:55:19.830 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-12-15T22:55:19.840 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-12-15T22:55:19.842 INFO:teuthology.orchestra.run.smithi072.stdout:======================================================================================= 2023-12-15T22:55:19.842 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2023-12-15T22:55:19.842 INFO:teuthology.orchestra.run.smithi072.stdout:======================================================================================= 2023-12-15T22:55:19.842 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2023-12-15T22:55:19.842 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-rook noarch 2:19.0.0-201.gcd750a7f.el9 ceph-noarch 50 k 2023-12-15T22:55:19.842 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2023-12-15T22:55:19.842 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-15T22:55:19.842 INFO:teuthology.orchestra.run.smithi072.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-15T22:55:19.842 INFO:teuthology.orchestra.run.smithi072.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-15T22:55:19.842 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-15T22:55:19.842 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-15T22:55:19.843 INFO:teuthology.orchestra.run.smithi072.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-15T22:55:19.843 INFO:teuthology.orchestra.run.smithi072.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-15T22:55:19.843 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-15T22:55:19.843 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-15T22:55:19.843 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-15T22:55:19.843 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-15T22:55:19.843 INFO:teuthology.orchestra.run.smithi072.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-15T22:55:19.843 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:19.843 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2023-12-15T22:55:19.843 INFO:teuthology.orchestra.run.smithi072.stdout:======================================================================================= 2023-12-15T22:55:19.843 INFO:teuthology.orchestra.run.smithi072.stdout:Install 13 Packages 2023-12-15T22:55:19.843 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:19.844 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 2.0 M 2023-12-15T22:55:19.844 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 26 M 2023-12-15T22:55:19.844 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2023-12-15T22:55:19.948 INFO:teuthology.orchestra.run.smithi071.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 664 kB/s | 79 kB 00:00 2023-12-15T22:55:20.011 INFO:teuthology.orchestra.run.smithi072.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 210 kB/s | 24 kB 00:00 2023-12-15T22:55:20.028 INFO:teuthology.orchestra.run.smithi072.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 136 kB/s | 18 kB 00:00 2023-12-15T22:55:20.070 INFO:teuthology.orchestra.run.smithi072.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.9 MB/s | 174 kB 00:00 2023-12-15T22:55:20.096 INFO:teuthology.orchestra.run.smithi072.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2023-12-15T22:55:20.131 INFO:teuthology.orchestra.run.smithi072.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.4 MB/s | 211 kB 00:00 2023-12-15T22:55:20.148 INFO:teuthology.orchestra.run.smithi071.stdout:(2/2): ceph-fuse-19.0.0-201.gcd750a7f.el9.x86_6 2.6 MB/s | 832 kB 00:00 2023-12-15T22:55:20.149 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-12-15T22:55:20.149 INFO:teuthology.orchestra.run.smithi071.stdout:Total 2.8 MB/s | 911 kB 00:00 2023-12-15T22:55:20.150 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-12-15T22:55:20.157 INFO:teuthology.orchestra.run.smithi072.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 707 kB/s | 43 kB 00:00 2023-12-15T22:55:20.170 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-12-15T22:55:20.170 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-12-15T22:55:20.199 INFO:teuthology.orchestra.run.smithi072.stdout:(7/13): ceph-mgr-rook-19.0.0-201.gcd750a7f.el9. 165 kB/s | 50 kB 00:00 2023-12-15T22:55:20.244 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-12-15T22:55:20.245 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-12-15T22:55:20.283 INFO:teuthology.orchestra.run.smithi072.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 119 kB/s | 15 kB 00:00 2023-12-15T22:55:20.308 INFO:teuthology.orchestra.run.smithi072.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 182 kB/s | 32 kB 00:00 2023-12-15T22:55:20.424 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-12-15T22:55:20.442 INFO:teuthology.orchestra.run.smithi072.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 442 kB/s | 59 kB 00:00 2023-12-15T22:55:20.476 INFO:teuthology.orchestra.run.smithi072.stdout:(11/13): python3-google-auth-2.23.4-1.el9.noarc 713 kB/s | 197 kB 00:00 2023-12-15T22:55:20.535 INFO:teuthology.orchestra.run.smithi072.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 974 kB/s | 90 kB 00:00 2023-12-15T22:55:20.575 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-12-15T22:55:20.577 INFO:teuthology.orchestra.run.smithi072.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.5 MB/s | 1.0 MB 00:00 2023-12-15T22:55:20.578 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2023-12-15T22:55:20.579 INFO:teuthology.orchestra.run.smithi072.stdout:Total 2.8 MB/s | 2.0 MB 00:00 2023-12-15T22:55:20.579 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2023-12-15T22:55:20.592 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2023-12-15T22:55:20.592 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2023-12-15T22:55:20.640 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-fuse-2:19.0.0-201.gcd750a7f.el9.x86_64 2/2 2023-12-15T22:55:20.687 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2023-12-15T22:55:20.688 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2023-12-15T22:55:21.200 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2023-12-15T22:55:21.357 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-fuse-2:19.0.0-201.gcd750a7f.el9.x86_64 2/2 2023-12-15T22:55:21.357 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-fuse-2:19.0.0-201.gcd750a7f.el9.x86_64 1/2 2023-12-15T22:55:21.357 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-12-15T22:55:21.413 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-12-15T22:55:21.479 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-12-15T22:55:21.526 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-12-15T22:55:21.568 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-12-15T22:55:21.661 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-12-15T22:55:21.688 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-12-15T22:55:21.689 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:21.689 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-12-15T22:55:21.689 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-fuse-2:19.0.0-201.gcd750a7f.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-12-15T22:55:21.689 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:21.689 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-12-15T22:55:21.770 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-12-15T22:55:21.831 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-12-15T22:55:21.960 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-volume 2023-12-15T22:55:22.531 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:58 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:55:22.562 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-12-15T22:55:22.623 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-12-15T22:55:22.660 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-12-15T22:55:22.677 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-12-15T22:55:22.678 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-12-15T22:55:22.678 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-12-15T22:55:22.678 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-12-15T22:55:22.678 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-12-15T22:55:22.678 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-volume noarch 2:19.0.0-201.gcd750a7f.el9 ceph-noarch 262 k 2023-12-15T22:55:22.678 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:22.678 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-12-15T22:55:22.678 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-12-15T22:55:22.678 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2023-12-15T22:55:22.678 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:22.679 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 262 k 2023-12-15T22:55:22.679 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 1.2 M 2023-12-15T22:55:22.679 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-12-15T22:55:22.707 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-12-15T22:55:22.739 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-rook-2:19.0.0-201.gcd750a7f.el9.noarch 13/13 2023-12-15T22:55:23.069 INFO:teuthology.orchestra.run.smithi071.stdout:ceph-volume-19.0.0-201.gcd750a7f.el9.noarch.rpm 674 kB/s | 262 kB 00:00 2023-12-15T22:55:23.069 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-12-15T22:55:23.070 INFO:teuthology.orchestra.run.smithi071.stdout:Total 671 kB/s | 262 kB 00:00 2023-12-15T22:55:23.070 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-12-15T22:55:23.078 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-12-15T22:55:23.078 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-12-15T22:55:23.116 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-12-15T22:55:23.117 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-12-15T22:55:23.333 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-12-15T22:55:23.404 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-volume-2:19.0.0-201.gcd750a7f.el9.noarch 1/1 2023-12-15T22:55:23.420 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-volume-2:19.0.0-201.gcd750a7f.el9.noarch 1/1 2023-12-15T22:55:23.420 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-15T22:55:23.420 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-12-15T22:55:23.420 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:23.737 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-201.gcd750a7f.el9.noarch 13/13 2023-12-15T22:55:23.737 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-rook-2:19.0.0-201.gcd750a7f.el9.noarch 1/13 2023-12-15T22:55:23.738 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-12-15T22:55:23.738 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-12-15T22:55:23.738 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-12-15T22:55:23.738 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-12-15T22:55:23.738 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-12-15T22:55:23.738 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-12-15T22:55:23.738 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-12-15T22:55:23.738 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-12-15T22:55:23.738 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-12-15T22:55:23.738 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-12-15T22:55:23.738 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-12-15T22:55:24.114 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-12-15T22:55:24.115 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:24.115 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2023-12-15T22:55:24.115 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-rook-2:19.0.0-201.gcd750a7f.el9.noarch 2023-12-15T22:55:24.115 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-15T22:55:24.115 INFO:teuthology.orchestra.run.smithi072.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-15T22:55:24.115 INFO:teuthology.orchestra.run.smithi072.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-15T22:55:24.115 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-15T22:55:24.115 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-15T22:55:24.115 INFO:teuthology.orchestra.run.smithi072.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-15T22:55:24.115 INFO:teuthology.orchestra.run.smithi072.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-15T22:55:24.115 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-15T22:55:24.115 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-15T22:55:24.115 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-15T22:55:24.115 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-15T22:55:24.116 INFO:teuthology.orchestra.run.smithi072.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-15T22:55:24.116 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:24.116 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-12-15T22:55:24.414 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-mgr-cephadm 2023-12-15T22:55:24.476 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-volume-2:19.0.0-201.gcd750a7f.el9.noarch 1/1 2023-12-15T22:55:24.476 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:24.476 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-12-15T22:55:24.476 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-volume-2:19.0.0-201.gcd750a7f.el9.noarch 2023-12-15T22:55:24.476 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:24.476 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-12-15T22:55:24.804 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librados-devel 2023-12-15T22:55:24.975 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:05:00 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:55:25.121 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-12-15T22:55:25.122 INFO:teuthology.orchestra.run.smithi072.stdout:======================================================================================= 2023-12-15T22:55:25.122 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2023-12-15T22:55:25.122 INFO:teuthology.orchestra.run.smithi072.stdout:======================================================================================= 2023-12-15T22:55:25.122 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2023-12-15T22:55:25.122 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-cephadm noarch 2:19.0.0-201.gcd750a7f.el9 ceph-noarch 133 k 2023-12-15T22:55:25.122 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2023-12-15T22:55:25.122 INFO:teuthology.orchestra.run.smithi072.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-15T22:55:25.122 INFO:teuthology.orchestra.run.smithi072.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-15T22:55:25.122 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-15T22:55:25.122 INFO:teuthology.orchestra.run.smithi072.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-15T22:55:25.123 INFO:teuthology.orchestra.run.smithi072.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-15T22:55:25.123 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:25.123 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2023-12-15T22:55:25.123 INFO:teuthology.orchestra.run.smithi072.stdout:======================================================================================= 2023-12-15T22:55:25.123 INFO:teuthology.orchestra.run.smithi072.stdout:Install 6 Packages 2023-12-15T22:55:25.123 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:25.123 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 6.8 M 2023-12-15T22:55:25.123 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 33 M 2023-12-15T22:55:25.123 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2023-12-15T22:55:25.361 INFO:teuthology.orchestra.run.smithi072.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.4 MB/s | 228 kB 00:00 2023-12-15T22:55:25.378 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:05:01 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:55:25.486 INFO:teuthology.orchestra.run.smithi072.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2023-12-15T22:55:25.523 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-12-15T22:55:25.523 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-12-15T22:55:25.523 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2023-12-15T22:55:25.523 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-12-15T22:55:25.523 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-12-15T22:55:25.523 INFO:teuthology.orchestra.run.smithi071.stdout: librados-devel x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 127 k 2023-12-15T22:55:25.524 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:25.524 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-12-15T22:55:25.524 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-12-15T22:55:25.524 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2023-12-15T22:55:25.524 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:25.524 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 127 k 2023-12-15T22:55:25.524 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 460 k 2023-12-15T22:55:25.524 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-12-15T22:55:25.580 INFO:teuthology.orchestra.run.smithi072.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 625 kB/s | 58 kB 00:00 2023-12-15T22:55:25.622 INFO:teuthology.orchestra.run.smithi072.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 1.2 MB/s | 51 kB 00:00 2023-12-15T22:55:25.656 INFO:teuthology.orchestra.run.smithi072.stdout:(5/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.8 MB/s | 548 kB 00:00 2023-12-15T22:55:25.998 INFO:teuthology.orchestra.run.smithi072.stdout:(6/6): ceph-mgr-cephadm-19.0.0-201.gcd750a7f.el 166 kB/s | 133 kB 00:00 2023-12-15T22:55:26.000 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2023-12-15T22:55:26.000 INFO:teuthology.orchestra.run.smithi072.stdout:Total 7.7 MB/s | 6.8 MB 00:00 2023-12-15T22:55:26.000 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2023-12-15T22:55:26.010 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2023-12-15T22:55:26.010 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2023-12-15T22:55:26.049 INFO:teuthology.orchestra.run.smithi071.stdout:librados-devel-19.0.0-201.gcd750a7f.el9.x86_64. 242 kB/s | 127 kB 00:00 2023-12-15T22:55:26.049 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-12-15T22:55:26.050 INFO:teuthology.orchestra.run.smithi071.stdout:Total 241 kB/s | 127 kB 00:00 2023-12-15T22:55:26.050 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-12-15T22:55:26.055 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-12-15T22:55:26.055 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-12-15T22:55:26.099 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2023-12-15T22:55:26.099 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2023-12-15T22:55:26.112 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-12-15T22:55:26.112 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-12-15T22:55:26.251 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-12-15T22:55:26.371 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librados-devel-2:19.0.0-201.gcd750a7f.el9.x86_64 1/1 2023-12-15T22:55:26.384 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2023-12-15T22:55:26.550 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-12-15T22:55:26.610 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-12-15T22:55:26.817 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados-devel-2:19.0.0-201.gcd750a7f.el9.x86_64 1/1 2023-12-15T22:55:27.128 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados-devel-2:19.0.0-201.gcd750a7f.el9.x86_64 1/1 2023-12-15T22:55:27.128 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:27.128 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-12-15T22:55:27.128 INFO:teuthology.orchestra.run.smithi071.stdout: librados-devel-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:55:27.128 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:27.128 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-12-15T22:55:27.156 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-12-15T22:55:27.250 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-12-15T22:55:27.319 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-12-15T22:55:27.352 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-201.gcd750a7f.el9.noarch 6/6 2023-12-15T22:55:27.389 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install libcephfs2 2023-12-15T22:55:27.952 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:05:03 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:55:28.049 INFO:teuthology.orchestra.run.smithi071.stdout:Package libcephfs2-2:19.0.0-201.gcd750a7f.el9.x86_64 is already installed. 2023-12-15T22:55:28.099 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-12-15T22:55:28.100 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2023-12-15T22:55:28.101 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-12-15T22:55:28.165 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install libcephfs-devel 2023-12-15T22:55:28.483 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-201.gcd750a7f.el9.noarch 6/6 2023-12-15T22:55:28.483 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-201.gcd750a7f.el9.noarch 1/6 2023-12-15T22:55:28.483 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-12-15T22:55:28.483 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-12-15T22:55:28.483 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-12-15T22:55:28.483 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-12-15T22:55:28.729 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:05:04 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:55:28.791 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-12-15T22:55:28.791 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:28.791 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2023-12-15T22:55:28.792 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-cephadm-2:19.0.0-201.gcd750a7f.el9.noarch 2023-12-15T22:55:28.792 INFO:teuthology.orchestra.run.smithi072.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-15T22:55:28.792 INFO:teuthology.orchestra.run.smithi072.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-15T22:55:28.792 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-15T22:55:28.792 INFO:teuthology.orchestra.run.smithi072.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-15T22:55:28.792 INFO:teuthology.orchestra.run.smithi072.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-15T22:55:28.792 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:28.792 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-12-15T22:55:28.874 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-12-15T22:55:28.875 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-12-15T22:55:28.875 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2023-12-15T22:55:28.875 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-12-15T22:55:28.875 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-12-15T22:55:28.875 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs-devel x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 31 k 2023-12-15T22:55:28.875 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:28.875 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-12-15T22:55:28.875 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-12-15T22:55:28.875 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2023-12-15T22:55:28.875 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:28.876 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 31 k 2023-12-15T22:55:28.876 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 140 k 2023-12-15T22:55:28.876 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-12-15T22:55:29.108 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-fuse 2023-12-15T22:55:29.196 INFO:teuthology.orchestra.run.smithi071.stdout:libcephfs-devel-19.0.0-201.gcd750a7f.el9.x86_64 97 kB/s | 31 kB 00:00 2023-12-15T22:55:29.197 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-12-15T22:55:29.198 INFO:teuthology.orchestra.run.smithi071.stdout:Total 97 kB/s | 31 kB 00:00 2023-12-15T22:55:29.198 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-12-15T22:55:29.201 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-12-15T22:55:29.202 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-12-15T22:55:29.233 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-12-15T22:55:29.234 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-12-15T22:55:29.325 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-12-15T22:55:29.442 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephfs-devel-2:19.0.0-201.gcd750a7f.el9.x86_64 1/1 2023-12-15T22:55:29.663 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:05:05 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:55:29.807 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-12-15T22:55:29.808 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-12-15T22:55:29.808 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2023-12-15T22:55:29.808 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-12-15T22:55:29.808 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2023-12-15T22:55:29.808 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-fuse x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 832 k 2023-12-15T22:55:29.808 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2023-12-15T22:55:29.808 INFO:teuthology.orchestra.run.smithi072.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-15T22:55:29.808 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:29.809 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2023-12-15T22:55:29.809 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-12-15T22:55:29.809 INFO:teuthology.orchestra.run.smithi072.stdout:Install 2 Packages 2023-12-15T22:55:29.809 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:29.809 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 911 k 2023-12-15T22:55:29.809 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 3.0 M 2023-12-15T22:55:29.809 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2023-12-15T22:55:29.863 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-201.gcd750a7f.el9.x86_64 1/1 2023-12-15T22:55:29.914 INFO:teuthology.orchestra.run.smithi072.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 750 kB/s | 79 kB 00:00 2023-12-15T22:55:30.115 INFO:teuthology.orchestra.run.smithi072.stdout:(2/2): ceph-fuse-19.0.0-201.gcd750a7f.el9.x86_6 2.7 MB/s | 832 kB 00:00 2023-12-15T22:55:30.116 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2023-12-15T22:55:30.116 INFO:teuthology.orchestra.run.smithi072.stdout:Total 2.9 MB/s | 911 kB 00:00 2023-12-15T22:55:30.116 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2023-12-15T22:55:30.135 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephfs-devel-2:19.0.0-201.gcd750a7f.el9.x86_64 1/1 2023-12-15T22:55:30.135 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:30.135 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-12-15T22:55:30.135 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs-devel-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:55:30.135 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:30.135 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-12-15T22:55:30.137 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2023-12-15T22:55:30.137 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2023-12-15T22:55:30.211 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2023-12-15T22:55:30.212 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2023-12-15T22:55:30.315 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librados2 2023-12-15T22:55:30.393 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2023-12-15T22:55:30.538 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-12-15T22:55:30.617 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-fuse-2:19.0.0-201.gcd750a7f.el9.x86_64 2/2 2023-12-15T22:55:30.865 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:05:06 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:55:30.958 INFO:teuthology.orchestra.run.smithi071.stdout:Package librados2-2:19.0.0-201.gcd750a7f.el9.x86_64 is already installed. 2023-12-15T22:55:31.009 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-12-15T22:55:31.010 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2023-12-15T22:55:31.010 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-12-15T22:55:31.073 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librbd1 2023-12-15T22:55:31.343 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-fuse-2:19.0.0-201.gcd750a7f.el9.x86_64 2/2 2023-12-15T22:55:31.343 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-fuse-2:19.0.0-201.gcd750a7f.el9.x86_64 1/2 2023-12-15T22:55:31.629 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:05:07 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:55:31.653 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-12-15T22:55:31.653 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:31.653 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2023-12-15T22:55:31.653 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-fuse-2:19.0.0-201.gcd750a7f.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-12-15T22:55:31.653 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:31.653 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-12-15T22:55:31.723 INFO:teuthology.orchestra.run.smithi071.stdout:Package librbd1-2:19.0.0-201.gcd750a7f.el9.x86_64 is already installed. 2023-12-15T22:55:31.774 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-12-15T22:55:31.775 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2023-12-15T22:55:31.775 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-12-15T22:55:31.841 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rados 2023-12-15T22:55:31.928 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-volume 2023-12-15T22:55:32.392 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:05:08 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:55:32.484 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rados-2:19.0.0-201.gcd750a7f.el9.x86_64 is already installed. 2023-12-15T22:55:32.494 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:05:08 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:55:32.534 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-12-15T22:55:32.535 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2023-12-15T22:55:32.535 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-12-15T22:55:32.606 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rgw 2023-12-15T22:55:32.645 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-12-15T22:55:32.645 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-12-15T22:55:32.646 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2023-12-15T22:55:32.646 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-12-15T22:55:32.646 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2023-12-15T22:55:32.646 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-volume noarch 2:19.0.0-201.gcd750a7f.el9 ceph-noarch 262 k 2023-12-15T22:55:32.646 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:32.646 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2023-12-15T22:55:32.646 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-12-15T22:55:32.646 INFO:teuthology.orchestra.run.smithi072.stdout:Install 1 Package 2023-12-15T22:55:32.646 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:32.646 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 262 k 2023-12-15T22:55:32.646 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 1.2 M 2023-12-15T22:55:32.646 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2023-12-15T22:55:33.047 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-volume-19.0.0-201.gcd750a7f.el9.noarch.rpm 656 kB/s | 262 kB 00:00 2023-12-15T22:55:33.048 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2023-12-15T22:55:33.048 INFO:teuthology.orchestra.run.smithi072.stdout:Total 653 kB/s | 262 kB 00:00 2023-12-15T22:55:33.048 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2023-12-15T22:55:33.056 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2023-12-15T22:55:33.056 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2023-12-15T22:55:33.093 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2023-12-15T22:55:33.093 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2023-12-15T22:55:33.163 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:05:09 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:55:33.256 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rgw-2:19.0.0-201.gcd750a7f.el9.x86_64 is already installed. 2023-12-15T22:55:33.302 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2023-12-15T22:55:33.308 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-12-15T22:55:33.309 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2023-12-15T22:55:33.309 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-12-15T22:55:33.375 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-volume-2:19.0.0-201.gcd750a7f.el9.noarch 1/1 2023-12-15T22:55:33.378 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-cephfs 2023-12-15T22:55:33.391 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-volume-2:19.0.0-201.gcd750a7f.el9.noarch 1/1 2023-12-15T22:55:33.391 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-15T22:55:33.391 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-12-15T22:55:33.391 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:33.937 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:05:09 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:55:34.031 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-cephfs-2:19.0.0-201.gcd750a7f.el9.x86_64 is already installed. 2023-12-15T22:55:34.080 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-12-15T22:55:34.081 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2023-12-15T22:55:34.081 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-12-15T22:55:34.147 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rbd 2023-12-15T22:55:34.563 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-volume-2:19.0.0-201.gcd750a7f.el9.noarch 1/1 2023-12-15T22:55:34.563 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:34.563 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2023-12-15T22:55:34.563 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-volume-2:19.0.0-201.gcd750a7f.el9.noarch 2023-12-15T22:55:34.563 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:34.563 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-12-15T22:55:34.700 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:05:10 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:55:34.794 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rbd-2:19.0.0-201.gcd750a7f.el9.x86_64 is already installed. 2023-12-15T22:55:34.844 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-12-15T22:55:34.845 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2023-12-15T22:55:34.845 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-12-15T22:55:34.912 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-fuse 2023-12-15T22:55:34.963 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install librados-devel 2023-12-15T22:55:35.465 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:05:11 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:55:35.530 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:05:11 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:55:35.608 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-12-15T22:55:35.608 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-12-15T22:55:35.608 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2023-12-15T22:55:35.608 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-12-15T22:55:35.608 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-12-15T22:55:35.608 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-fuse x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 86 k 2023-12-15T22:55:35.608 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:35.609 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-12-15T22:55:35.609 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-12-15T22:55:35.609 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2023-12-15T22:55:35.609 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:35.609 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 86 k 2023-12-15T22:55:35.609 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 256 k 2023-12-15T22:55:35.609 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-12-15T22:55:35.674 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-12-15T22:55:35.674 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-12-15T22:55:35.674 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repo Size 2023-12-15T22:55:35.674 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-12-15T22:55:35.674 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2023-12-15T22:55:35.674 INFO:teuthology.orchestra.run.smithi072.stdout: librados-devel x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 127 k 2023-12-15T22:55:35.674 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:35.674 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2023-12-15T22:55:35.674 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-12-15T22:55:35.674 INFO:teuthology.orchestra.run.smithi072.stdout:Install 1 Package 2023-12-15T22:55:35.674 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:35.674 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 127 k 2023-12-15T22:55:35.674 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 460 k 2023-12-15T22:55:35.674 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2023-12-15T22:55:35.904 INFO:teuthology.orchestra.run.smithi072.stdout:librados-devel-19.0.0-201.gcd750a7f.el9.x86_64. 553 kB/s | 127 kB 00:00 2023-12-15T22:55:35.904 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2023-12-15T22:55:35.905 INFO:teuthology.orchestra.run.smithi072.stdout:Total 550 kB/s | 127 kB 00:00 2023-12-15T22:55:35.905 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2023-12-15T22:55:35.910 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2023-12-15T22:55:35.910 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2023-12-15T22:55:35.929 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-fuse-19.0.0-201.gcd750a7f.el9.x86_64.rpm 268 kB/s | 86 kB 00:00 2023-12-15T22:55:35.930 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-12-15T22:55:35.930 INFO:teuthology.orchestra.run.smithi071.stdout:Total 267 kB/s | 86 kB 00:00 2023-12-15T22:55:35.930 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-12-15T22:55:35.937 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-12-15T22:55:35.937 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-12-15T22:55:35.965 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2023-12-15T22:55:35.965 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2023-12-15T22:55:35.991 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-12-15T22:55:35.991 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-12-15T22:55:36.107 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2023-12-15T22:55:36.127 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-12-15T22:55:36.211 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librados-devel-2:19.0.0-201.gcd750a7f.el9.x86_64 1/1 2023-12-15T22:55:36.227 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-fuse-2:19.0.0-201.gcd750a7f.el9.x86_64 1/1 2023-12-15T22:55:36.617 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-fuse-2:19.0.0-201.gcd750a7f.el9.x86_64 1/1 2023-12-15T22:55:36.689 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librados-devel-2:19.0.0-201.gcd750a7f.el9.x86_64 1/1 2023-12-15T22:55:36.927 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-fuse-2:19.0.0-201.gcd750a7f.el9.x86_64 1/1 2023-12-15T22:55:36.928 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:36.928 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-12-15T22:55:36.928 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-fuse-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:55:36.928 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:36.928 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-12-15T22:55:36.994 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librados-devel-2:19.0.0-201.gcd750a7f.el9.x86_64 1/1 2023-12-15T22:55:36.994 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:36.994 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2023-12-15T22:55:36.994 INFO:teuthology.orchestra.run.smithi072.stdout: librados-devel-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:55:36.994 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:36.994 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-12-15T22:55:37.171 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-mirror 2023-12-15T22:55:37.280 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install libcephfs2 2023-12-15T22:55:37.734 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:05:13 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:55:37.842 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:05:13 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:55:37.881 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-12-15T22:55:37.881 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-12-15T22:55:37.882 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2023-12-15T22:55:37.882 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-12-15T22:55:37.882 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-12-15T22:55:37.882 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-mirror x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 3.0 M 2023-12-15T22:55:37.882 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:37.882 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-12-15T22:55:37.882 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-12-15T22:55:37.882 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2023-12-15T22:55:37.882 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:37.883 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 3.0 M 2023-12-15T22:55:37.883 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 13 M 2023-12-15T22:55:37.883 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-12-15T22:55:37.939 INFO:teuthology.orchestra.run.smithi072.stdout:Package libcephfs2-2:19.0.0-201.gcd750a7f.el9.x86_64 is already installed. 2023-12-15T22:55:37.992 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-12-15T22:55:37.993 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2023-12-15T22:55:37.993 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-12-15T22:55:38.065 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install libcephfs-devel 2023-12-15T22:55:38.466 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-mirror-19.0.0-201.gcd750a7f.el9.x86_64.rpm 5.1 MB/s | 3.0 MB 00:00 2023-12-15T22:55:38.466 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-12-15T22:55:38.467 INFO:teuthology.orchestra.run.smithi071.stdout:Total 5.1 MB/s | 3.0 MB 00:00 2023-12-15T22:55:38.467 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-12-15T22:55:38.475 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-12-15T22:55:38.476 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-12-15T22:55:38.547 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-12-15T22:55:38.548 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-12-15T22:55:38.630 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:05:14 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:55:38.796 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-12-15T22:55:38.796 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-12-15T22:55:38.796 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repo Size 2023-12-15T22:55:38.796 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-12-15T22:55:38.796 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2023-12-15T22:55:38.796 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs-devel x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 31 k 2023-12-15T22:55:38.796 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:38.796 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2023-12-15T22:55:38.797 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-12-15T22:55:38.797 INFO:teuthology.orchestra.run.smithi072.stdout:Install 1 Package 2023-12-15T22:55:38.797 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:38.797 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 31 k 2023-12-15T22:55:38.797 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 140 k 2023-12-15T22:55:38.797 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2023-12-15T22:55:38.978 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-12-15T22:55:39.049 INFO:teuthology.orchestra.run.smithi072.stdout:libcephfs-devel-19.0.0-201.gcd750a7f.el9.x86_64 123 kB/s | 31 kB 00:00 2023-12-15T22:55:39.050 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-mirror-2:19.0.0-201.gcd750a7f.el9.x86_64 1/1 2023-12-15T22:55:39.050 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2023-12-15T22:55:39.050 INFO:teuthology.orchestra.run.smithi072.stdout:Total 123 kB/s | 31 kB 00:00 2023-12-15T22:55:39.050 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2023-12-15T22:55:39.054 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2023-12-15T22:55:39.054 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2023-12-15T22:55:39.074 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-mirror-2:19.0.0-201.gcd750a7f.el9.x86_64 1/1 2023-12-15T22:55:39.075 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-15T22:55:39.075 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-15T22:55:39.075 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-15T22:55:39.075 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-15T22:55:39.075 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:39.084 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2023-12-15T22:55:39.084 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2023-12-15T22:55:39.167 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2023-12-15T22:55:39.289 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libcephfs-devel-2:19.0.0-201.gcd750a7f.el9.x86_64 1/1 2023-12-15T22:55:39.713 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-201.gcd750a7f.el9.x86_64 1/1 2023-12-15T22:55:39.983 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libcephfs-devel-2:19.0.0-201.gcd750a7f.el9.x86_64 1/1 2023-12-15T22:55:39.983 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:39.983 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2023-12-15T22:55:39.983 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs-devel-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:55:39.983 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:39.983 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-12-15T22:55:40.193 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install librados2 2023-12-15T22:55:40.219 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-mirror-2:19.0.0-201.gcd750a7f.el9.x86_64 1/1 2023-12-15T22:55:40.219 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:40.219 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-12-15T22:55:40.220 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-mirror-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:55:40.220 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:40.220 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-12-15T22:55:40.452 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-nbd 2023-12-15T22:55:40.747 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:05:16 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:55:40.840 INFO:teuthology.orchestra.run.smithi072.stdout:Package librados2-2:19.0.0-201.gcd750a7f.el9.x86_64 is already installed. 2023-12-15T22:55:40.890 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-12-15T22:55:40.891 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2023-12-15T22:55:40.891 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-12-15T22:55:40.963 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install librbd1 2023-12-15T22:55:41.019 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:05:17 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:55:41.162 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-12-15T22:55:41.163 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-12-15T22:55:41.163 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2023-12-15T22:55:41.163 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-12-15T22:55:41.163 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-12-15T22:55:41.164 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-nbd x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 169 k 2023-12-15T22:55:41.164 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:41.164 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-12-15T22:55:41.164 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-12-15T22:55:41.164 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2023-12-15T22:55:41.164 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:41.164 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 169 k 2023-12-15T22:55:41.164 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 537 k 2023-12-15T22:55:41.164 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-12-15T22:55:41.517 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:05:17 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:55:41.544 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-nbd-19.0.0-201.gcd750a7f.el9.x86_64.rpm 446 kB/s | 169 kB 00:00 2023-12-15T22:55:41.544 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-12-15T22:55:41.545 INFO:teuthology.orchestra.run.smithi071.stdout:Total 444 kB/s | 169 kB 00:00 2023-12-15T22:55:41.545 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-12-15T22:55:41.553 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-12-15T22:55:41.553 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-12-15T22:55:41.607 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-12-15T22:55:41.608 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-12-15T22:55:41.610 INFO:teuthology.orchestra.run.smithi072.stdout:Package librbd1-2:19.0.0-201.gcd750a7f.el9.x86_64 is already installed. 2023-12-15T22:55:41.660 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-12-15T22:55:41.661 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2023-12-15T22:55:41.661 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-12-15T22:55:41.741 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install python3-rados 2023-12-15T22:55:41.754 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-12-15T22:55:41.851 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-nbd-2:19.0.0-201.gcd750a7f.el9.x86_64 1/1 2023-12-15T22:55:42.288 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:05:18 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:55:42.314 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-nbd-2:19.0.0-201.gcd750a7f.el9.x86_64 1/1 2023-12-15T22:55:42.382 INFO:teuthology.orchestra.run.smithi072.stdout:Package python3-rados-2:19.0.0-201.gcd750a7f.el9.x86_64 is already installed. 2023-12-15T22:55:42.432 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-12-15T22:55:42.433 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2023-12-15T22:55:42.433 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-12-15T22:55:42.510 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install python3-rgw 2023-12-15T22:55:42.608 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-nbd-2:19.0.0-201.gcd750a7f.el9.x86_64 1/1 2023-12-15T22:55:42.608 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:42.608 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-12-15T22:55:42.608 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-nbd-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:55:42.608 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:42.608 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-12-15T22:55:42.864 DEBUG:teuthology.parallel:result is None 2023-12-15T22:55:43.064 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:05:19 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:55:43.157 INFO:teuthology.orchestra.run.smithi072.stdout:Package python3-rgw-2:19.0.0-201.gcd750a7f.el9.x86_64 is already installed. 2023-12-15T22:55:43.208 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-12-15T22:55:43.209 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2023-12-15T22:55:43.209 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-12-15T22:55:43.287 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install python3-cephfs 2023-12-15T22:55:43.843 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:05:19 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:55:43.937 INFO:teuthology.orchestra.run.smithi072.stdout:Package python3-cephfs-2:19.0.0-201.gcd750a7f.el9.x86_64 is already installed. 2023-12-15T22:55:43.986 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-12-15T22:55:43.987 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2023-12-15T22:55:43.987 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-12-15T22:55:44.064 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install python3-rbd 2023-12-15T22:55:44.617 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:05:20 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:55:44.710 INFO:teuthology.orchestra.run.smithi072.stdout:Package python3-rbd-2:19.0.0-201.gcd750a7f.el9.x86_64 is already installed. 2023-12-15T22:55:44.760 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-12-15T22:55:44.762 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2023-12-15T22:55:44.762 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-12-15T22:55:44.837 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install rbd-fuse 2023-12-15T22:55:45.395 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:05:21 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:55:45.539 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-12-15T22:55:45.540 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-12-15T22:55:45.540 INFO:teuthology.orchestra.run.smithi072.stdout: Package Architecture Version Repository Size 2023-12-15T22:55:45.540 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-12-15T22:55:45.540 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2023-12-15T22:55:45.540 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-fuse x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 86 k 2023-12-15T22:55:45.541 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:45.541 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2023-12-15T22:55:45.541 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-12-15T22:55:45.541 INFO:teuthology.orchestra.run.smithi072.stdout:Install 1 Package 2023-12-15T22:55:45.541 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:45.541 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 86 k 2023-12-15T22:55:45.541 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 256 k 2023-12-15T22:55:45.541 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2023-12-15T22:55:45.966 INFO:teuthology.orchestra.run.smithi072.stdout:rbd-fuse-19.0.0-201.gcd750a7f.el9.x86_64.rpm 202 kB/s | 86 kB 00:00 2023-12-15T22:55:45.967 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2023-12-15T22:55:45.967 INFO:teuthology.orchestra.run.smithi072.stdout:Total 201 kB/s | 86 kB 00:00 2023-12-15T22:55:45.967 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2023-12-15T22:55:45.974 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2023-12-15T22:55:45.974 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2023-12-15T22:55:46.030 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2023-12-15T22:55:46.030 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2023-12-15T22:55:46.170 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2023-12-15T22:55:46.282 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : rbd-fuse-2:19.0.0-201.gcd750a7f.el9.x86_64 1/1 2023-12-15T22:55:46.710 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: rbd-fuse-2:19.0.0-201.gcd750a7f.el9.x86_64 1/1 2023-12-15T22:55:46.992 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : rbd-fuse-2:19.0.0-201.gcd750a7f.el9.x86_64 1/1 2023-12-15T22:55:46.992 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:46.992 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2023-12-15T22:55:46.993 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-fuse-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:55:46.993 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:46.993 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-12-15T22:55:47.258 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install rbd-mirror 2023-12-15T22:55:47.819 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:05:23 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:55:47.967 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-12-15T22:55:47.968 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-12-15T22:55:47.968 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repo Size 2023-12-15T22:55:47.969 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-12-15T22:55:47.969 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2023-12-15T22:55:47.969 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-mirror x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 3.0 M 2023-12-15T22:55:47.969 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:47.969 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2023-12-15T22:55:47.969 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-12-15T22:55:47.969 INFO:teuthology.orchestra.run.smithi072.stdout:Install 1 Package 2023-12-15T22:55:47.969 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:47.969 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 3.0 M 2023-12-15T22:55:47.969 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 13 M 2023-12-15T22:55:47.969 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2023-12-15T22:55:48.544 INFO:teuthology.orchestra.run.smithi072.stdout:rbd-mirror-19.0.0-201.gcd750a7f.el9.x86_64.rpm 5.2 MB/s | 3.0 MB 00:00 2023-12-15T22:55:48.544 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2023-12-15T22:55:48.545 INFO:teuthology.orchestra.run.smithi072.stdout:Total 5.2 MB/s | 3.0 MB 00:00 2023-12-15T22:55:48.545 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2023-12-15T22:55:48.554 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2023-12-15T22:55:48.554 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2023-12-15T22:55:48.625 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2023-12-15T22:55:48.626 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2023-12-15T22:55:49.043 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2023-12-15T22:55:49.104 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : rbd-mirror-2:19.0.0-201.gcd750a7f.el9.x86_64 1/1 2023-12-15T22:55:49.128 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: rbd-mirror-2:19.0.0-201.gcd750a7f.el9.x86_64 1/1 2023-12-15T22:55:49.128 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-15T22:55:49.128 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-15T22:55:49.128 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-15T22:55:49.128 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-15T22:55:49.128 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:50.442 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : rbd-mirror-2:19.0.0-201.gcd750a7f.el9.x86_64 1/1 2023-12-15T22:55:50.442 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:50.443 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2023-12-15T22:55:50.443 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-mirror-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:55:50.443 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:50.443 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-12-15T22:55:50.717 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install rbd-nbd 2023-12-15T22:55:51.287 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:05:27 ago on Fri 15 Dec 2023 10:50:24 PM UTC. 2023-12-15T22:55:51.431 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-12-15T22:55:51.432 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-12-15T22:55:51.432 INFO:teuthology.orchestra.run.smithi072.stdout: Package Architecture Version Repository Size 2023-12-15T22:55:51.432 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-12-15T22:55:51.432 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2023-12-15T22:55:51.432 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-nbd x86_64 2:19.0.0-201.gcd750a7f.el9 ceph 169 k 2023-12-15T22:55:51.432 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:51.432 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2023-12-15T22:55:51.432 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-12-15T22:55:51.432 INFO:teuthology.orchestra.run.smithi072.stdout:Install 1 Package 2023-12-15T22:55:51.432 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:51.433 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 169 k 2023-12-15T22:55:51.433 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 537 k 2023-12-15T22:55:51.433 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2023-12-15T22:55:51.780 INFO:teuthology.orchestra.run.smithi072.stdout:rbd-nbd-19.0.0-201.gcd750a7f.el9.x86_64.rpm 488 kB/s | 169 kB 00:00 2023-12-15T22:55:51.780 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2023-12-15T22:55:51.780 INFO:teuthology.orchestra.run.smithi072.stdout:Total 487 kB/s | 169 kB 00:00 2023-12-15T22:55:51.780 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2023-12-15T22:55:51.788 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2023-12-15T22:55:51.788 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2023-12-15T22:55:51.843 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2023-12-15T22:55:51.843 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2023-12-15T22:55:51.987 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2023-12-15T22:55:52.105 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : rbd-nbd-2:19.0.0-201.gcd750a7f.el9.x86_64 1/1 2023-12-15T22:55:52.576 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: rbd-nbd-2:19.0.0-201.gcd750a7f.el9.x86_64 1/1 2023-12-15T22:55:52.887 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : rbd-nbd-2:19.0.0-201.gcd750a7f.el9.x86_64 1/1 2023-12-15T22:55:52.887 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:52.887 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2023-12-15T22:55:52.887 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-nbd-2:19.0.0-201.gcd750a7f.el9.x86_64 2023-12-15T22:55:52.887 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:52.887 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-12-15T22:55:53.165 DEBUG:teuthology.parallel:result is None 2023-12-15T22:55:53.165 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=cd750a7fc894b5dea3d055d595c0da884f63f94f 2023-12-15T22:55:53.386 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-15T22:55:53.410 INFO:teuthology.orchestra.run.smithi071.stdout:19.0.0-201.gcd750a7f.el9 2023-12-15T22:55:53.411 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-201.gcd750a7f.el9 2023-12-15T22:55:53.411 INFO:teuthology.task.install:The correct ceph version 19.0.0-201.gcd750a7f is installed. 2023-12-15T22:55:53.412 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=cd750a7fc894b5dea3d055d595c0da884f63f94f 2023-12-15T22:55:53.661 DEBUG:teuthology.orchestra.run.smithi072:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-15T22:55:53.682 INFO:teuthology.orchestra.run.smithi072.stdout:19.0.0-201.gcd750a7f.el9 2023-12-15T22:55:53.682 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-201.gcd750a7f.el9 2023-12-15T22:55:53.682 INFO:teuthology.task.install:The correct ceph version 19.0.0-201.gcd750a7f is installed. 2023-12-15T22:55:53.684 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-15T22:55:53.684 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-12-15T22:55:53.684 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-15T22:55:53.717 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2023-12-15T22:55:53.717 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-15T22:55:53.749 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-15T22:55:53.750 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-12-15T22:55:53.750 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/daemon-helper 2023-12-15T22:55:53.787 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-15T22:55:53.856 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2023-12-15T22:55:53.856 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/usr/bin/daemon-helper 2023-12-15T22:55:53.882 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-15T22:55:53.949 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-15T22:55:53.949 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-12-15T22:55:53.949 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-15T22:55:53.978 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-15T22:55:54.044 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2023-12-15T22:55:54.045 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-15T22:55:54.071 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-15T22:55:54.137 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-15T22:55:54.137 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-12-15T22:55:54.137 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/stdin-killer 2023-12-15T22:55:54.164 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-15T22:55:54.231 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2023-12-15T22:55:54.231 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/usr/bin/stdin-killer 2023-12-15T22:55:54.257 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-15T22:55:54.324 INFO:teuthology.run_tasks:Running task ceph... 2023-12-15T22:55:54.393 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-12-15T22:55:54.393 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 777 /var/log/ceph 2023-12-15T22:55:54.396 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 777 /var/log/ceph 2023-12-15T22:55:54.422 INFO:tasks.ceph:Disabling ceph logrotate... 2023-12-15T22:55:54.422 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-15T22:55:54.463 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-15T22:55:54.489 INFO:tasks.ceph:Creating extra log directories... 2023-12-15T22:55:54.489 DEBUG:teuthology.orchestra.run.smithi071:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-15T22:55:54.530 DEBUG:teuthology.orchestra.run.smithi072:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-15T22:55:54.566 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-12-15T22:55:54.566 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-15T22:55:54.566 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/sjust-2023-12-15_22:32:12-crimson-rados-wip-sjust-crimson-testing-2023-12-15-distro-default-smithi/7493355', 'branch': 'wip-sjust-crimson-testing-2023-12-15', 'description': 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects}', 'email': None, 'first_in_suite': False, 'job_id': '7493355', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'sjust-2023-12-15_22:32:12-crimson-rados-wip-sjust-crimson-testing-2023-12-15-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-sjust-crimson-testing-2023-12-15'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'sha1': 'cd750a7fc894b5dea3d055d595c0da884f63f94f', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': 'cd750a7fc894b5dea3d055d595c0da884f63f94f'}}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'wip-sjust-crimson-testing-2023-12-15', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'cd750a7fc894b5dea3d055d595c0da884f63f94f'}}, 'owner': 'scheduled_sjust@teuthology', 'priority': 51, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'], ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6485, 'sha1': 'cd750a7fc894b5dea3d055d595c0da884f63f94f', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'wip-sjust-crimson-testing-2023-12-15', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_cd750a7fc894b5dea3d055d595c0da884f63f94f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'cd750a7fc894b5dea3d055d595c0da884f63f94f', 'targets': {'smithi071.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAlS1MwrDgn45o6kUzFJ0kxTcv+jWab7PAGlEmLqOivn4AQC/AmL5RsWCnRViqkqngl33KaeYxO5ZMD18jIkhMY=', 'smithi072.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKMWi4rHWHdGiQVIsrAthgifQCjM9Y5hC8faCromrItnrKy59Pj4KasGyNB+439jYnGaYGZ5pFpQo2xCakbKqVY='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'sha1': 'cd750a7fc894b5dea3d055d595c0da884f63f94f', 'wait-for-scrub': False, 'cluster': 'ceph'}}, {'thrashosds': {'ceph_objectstore_tool': False, 'chance_down': 0, 'chance_inject_pause_short': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400}}, {'rados': {'clients': ['client.0'], 'objects': 50, 'op_weights': {'copy_from': 0, 'delete': 50, 'read': 100, 'rollback': 0, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73', 'timestamp': '2023-12-15_22:32:12', 'tube': 'smithi', 'user': 'sjust', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.1485078'} 2023-12-15T22:55:54.566 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-15T22:55:54.597 DEBUG:teuthology.orchestra.run.smithi072:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-15T22:55:54.611 DEBUG:teuthology.orchestra.run.smithi071:> sudo install -d -m0777 -- /var/run/ceph 2023-12-15T22:55:54.651 DEBUG:teuthology.orchestra.run.smithi072:> sudo install -d -m0777 -- /var/run/ceph 2023-12-15T22:55:54.678 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-12-15T22:55:54.679 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/scratch_devs of=/dev/stdout 2023-12-15T22:55:54.729 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-15T22:55:54.729 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_1 2023-12-15T22:55:54.784 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-15T22:55:54.784 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-15T22:55:54.784 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 663 Links: 1 2023-12-15T22:55:54.784 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-15T22:55:54.784 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2023-12-15T22:55:54.784 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2023-12-15 22:55:39.307078928 +0000 2023-12-15T22:55:54.784 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2023-12-15 22:49:32.118173682 +0000 2023-12-15T22:55:54.784 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2023-12-15 22:49:32.118173682 +0000 2023-12-15T22:55:54.784 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2023-12-15 22:49:32.118173682 +0000 2023-12-15T22:55:54.785 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-15T22:55:54.849 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2023-12-15T22:55:54.849 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2023-12-15T22:55:54.849 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000206195 s, 2.5 MB/s 2023-12-15T22:55:54.850 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-15T22:55:54.906 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_2 2023-12-15T22:55:54.960 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-15T22:55:54.960 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-15T22:55:54.960 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 682 Links: 1 2023-12-15T22:55:54.960 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-15T22:55:54.960 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2023-12-15T22:55:54.960 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2023-12-15 22:55:39.308078909 +0000 2023-12-15T22:55:54.960 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2023-12-15 22:49:32.624164117 +0000 2023-12-15T22:55:54.960 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2023-12-15 22:49:32.624164117 +0000 2023-12-15T22:55:54.961 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2023-12-15 22:49:32.624164117 +0000 2023-12-15T22:55:54.961 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-15T22:55:55.025 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2023-12-15T22:55:55.025 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2023-12-15T22:55:55.025 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000188793 s, 2.7 MB/s 2023-12-15T22:55:55.026 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-15T22:55:55.081 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_3 2023-12-15T22:55:55.136 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-15T22:55:55.136 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-15T22:55:55.136 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 700 Links: 1 2023-12-15T22:55:55.136 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-15T22:55:55.136 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2023-12-15T22:55:55.136 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2023-12-15 22:55:39.308078909 +0000 2023-12-15T22:55:55.136 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2023-12-15 22:49:33.134154476 +0000 2023-12-15T22:55:55.136 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2023-12-15 22:49:33.134154476 +0000 2023-12-15T22:55:55.136 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2023-12-15 22:49:33.134154476 +0000 2023-12-15T22:55:55.136 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-15T22:55:55.201 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2023-12-15T22:55:55.201 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2023-12-15T22:55:55.201 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000201702 s, 2.5 MB/s 2023-12-15T22:55:55.202 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-15T22:55:55.258 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_4 2023-12-15T22:55:55.312 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-15T22:55:55.312 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-15T22:55:55.312 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 721 Links: 1 2023-12-15T22:55:55.312 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-15T22:55:55.312 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2023-12-15T22:55:55.312 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2023-12-15 22:55:39.308078909 +0000 2023-12-15T22:55:55.312 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2023-12-15 22:49:33.605145572 +0000 2023-12-15T22:55:55.312 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2023-12-15 22:49:33.605145572 +0000 2023-12-15T22:55:55.312 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2023-12-15 22:49:33.605145572 +0000 2023-12-15T22:55:55.313 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-15T22:55:55.377 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2023-12-15T22:55:55.377 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2023-12-15T22:55:55.377 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000202474 s, 2.5 MB/s 2023-12-15T22:55:55.378 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-15T22:55:55.433 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2023-12-15T22:55:55.433 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2023-12-15T22:55:55.433 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/scratch_devs of=/dev/stdout 2023-12-15T22:55:55.449 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-15T22:55:55.450 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_1 2023-12-15T22:55:55.506 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-15T22:55:55.506 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-15T22:55:55.506 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 5h/5d Inode: 660 Links: 1 2023-12-15T22:55:55.506 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-15T22:55:55.506 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2023-12-15T22:55:55.506 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2023-12-15 22:55:49.364046744 +0000 2023-12-15T22:55:55.506 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2023-12-15 22:49:33.778317938 +0000 2023-12-15T22:55:55.506 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2023-12-15 22:49:33.778317938 +0000 2023-12-15T22:55:55.506 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2023-12-15 22:49:33.778317938 +0000 2023-12-15T22:55:55.506 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-15T22:55:55.572 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2023-12-15T22:55:55.572 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2023-12-15T22:55:55.572 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.000208363 s, 2.5 MB/s 2023-12-15T22:55:55.573 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-15T22:55:55.630 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_2 2023-12-15T22:55:55.685 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-15T22:55:55.685 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-15T22:55:55.685 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 5h/5d Inode: 680 Links: 1 2023-12-15T22:55:55.685 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-15T22:55:55.685 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2023-12-15T22:55:55.685 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2023-12-15 22:55:49.364046744 +0000 2023-12-15T22:55:55.686 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2023-12-15 22:49:34.270307707 +0000 2023-12-15T22:55:55.686 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2023-12-15 22:49:34.270307707 +0000 2023-12-15T22:55:55.686 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2023-12-15 22:49:34.270307707 +0000 2023-12-15T22:55:55.686 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-15T22:55:55.750 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2023-12-15T22:55:55.750 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2023-12-15T22:55:55.751 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.000192638 s, 2.7 MB/s 2023-12-15T22:55:55.752 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-15T22:55:55.807 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_3 2023-12-15T22:55:55.862 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-15T22:55:55.862 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-15T22:55:55.862 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 5h/5d Inode: 701 Links: 1 2023-12-15T22:55:55.862 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-15T22:55:55.862 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2023-12-15T22:55:55.862 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2023-12-15 22:55:49.365046723 +0000 2023-12-15T22:55:55.862 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2023-12-15 22:49:34.848295686 +0000 2023-12-15T22:55:55.862 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2023-12-15 22:49:34.848295686 +0000 2023-12-15T22:55:55.862 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2023-12-15 22:49:34.848295686 +0000 2023-12-15T22:55:55.863 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-15T22:55:55.927 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2023-12-15T22:55:55.927 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2023-12-15T22:55:55.927 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.000210903 s, 2.4 MB/s 2023-12-15T22:55:55.928 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-15T22:55:55.983 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_4 2023-12-15T22:55:56.037 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-15T22:55:56.037 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-15T22:55:56.037 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 5h/5d Inode: 719 Links: 1 2023-12-15T22:55:56.037 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-15T22:55:56.037 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2023-12-15T22:55:56.037 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2023-12-15 22:55:49.365046723 +0000 2023-12-15T22:55:56.037 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2023-12-15 22:49:35.321285849 +0000 2023-12-15T22:55:56.038 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2023-12-15 22:49:35.321285849 +0000 2023-12-15T22:55:56.038 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2023-12-15 22:49:35.321285849 +0000 2023-12-15T22:55:56.038 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-15T22:55:56.101 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2023-12-15T22:55:56.102 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2023-12-15T22:55:56.102 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.000177281 s, 2.9 MB/s 2023-12-15T22:55:56.103 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-15T22:55:56.158 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2023-12-15T22:55:56.158 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi071.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi072.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2023-12-15T22:55:56.158 INFO:tasks.ceph:Generating config... 2023-12-15T22:55:56.159 INFO:tasks.ceph:[osd] debug monc = 20 2023-12-15T22:55:56.159 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-15T22:55:56.159 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2023-12-15T22:55:56.159 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-12-15T22:55:56.159 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-15T22:55:56.159 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2023-12-15T22:55:56.159 INFO:tasks.ceph:[osd] crimson seastar smp = 3 2023-12-15T22:55:56.160 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-12-15T22:55:56.160 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-12-15T22:55:56.160 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-15T22:55:56.160 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-15T22:55:56.160 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-12-15T22:55:56.160 INFO:tasks.ceph:[osd] err to stderr = True 2023-12-15T22:55:56.160 INFO:tasks.ceph:[osd] log flush on exit = True 2023-12-15T22:55:56.160 INFO:tasks.ceph:[osd] log to file = False 2023-12-15T22:55:56.160 INFO:tasks.ceph:[osd] log to stderr = True 2023-12-15T22:55:56.160 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-15T22:55:56.160 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-15T22:55:56.160 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-15T22:55:56.160 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2023-12-15T22:55:56.160 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.3 2023-12-15T22:55:56.160 INFO:tasks.ceph:[osd] osd delete sleep = 1 2023-12-15T22:55:56.161 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-15T22:55:56.161 INFO:tasks.ceph:[osd] osd max backfills = 3 2023-12-15T22:55:56.161 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2023-12-15T22:55:56.161 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-15T22:55:56.161 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2023-12-15T22:55:56.161 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2023-12-15T22:55:56.161 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-12-15T22:55:56.161 INFO:tasks.ceph:[osd] osd snap trim sleep = 2 2023-12-15T22:55:56.161 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2023-12-15T22:55:56.161 INFO:tasks.ceph:[global] ms client mode = crc 2023-12-15T22:55:56.161 INFO:tasks.ceph:[global] ms cluster mode = crc 2023-12-15T22:55:56.161 INFO:tasks.ceph:[global] ms mon client mode = crc 2023-12-15T22:55:56.161 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2023-12-15T22:55:56.162 INFO:tasks.ceph:[global] ms mon service mode = crc 2023-12-15T22:55:56.162 INFO:tasks.ceph:[global] ms service mode = crc 2023-12-15T22:55:56.162 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2023-12-15T22:55:56.162 INFO:tasks.ceph:[global] osd_pool_default_size = 3 2023-12-15T22:55:56.162 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-15T22:55:56.162 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-15T22:55:56.162 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-15T22:55:56.162 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-15T22:55:56.162 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-15T22:55:56.162 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2023-12-15T22:55:56.162 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2023-12-15T22:55:56.162 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2023-12-15T22:55:56.162 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2023-12-15T22:55:56.162 INFO:tasks.ceph:[mon] osd pool default crimson = True 2023-12-15T22:55:56.162 INFO:tasks.ceph:[mon] paxos service trim min = 10 2023-12-15T22:55:56.162 INFO:tasks.ceph:Setting up mon.a... 2023-12-15T22:55:56.163 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-12-15T22:55:56.225 INFO:teuthology.orchestra.run.smithi071.stdout:creating /etc/ceph/ceph.keyring 2023-12-15T22:55:56.228 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-12-15T22:55:56.279 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-15T22:55:56.305 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.71')] 2023-12-15T22:55:56.306 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.71', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2023-12-15T22:55:56.306 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-12-15T22:55:56.306 DEBUG:teuthology.orchestra.run.smithi071:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-15T22:55:56.360 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.71 --print /home/ubuntu/cephtest/ceph.monmap 2023-12-15T22:55:56.446 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:55:56.446+0000 7f205d236e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:55:56.446 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:55:56.446+0000 7f205d236e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:55:56.446 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:55:56.446+0000 7f205d236e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:55:56.446 INFO:teuthology.orchestra.run.smithi071.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-12-15T22:55:56.447 INFO:teuthology.orchestra.run.smithi071.stdout:monmaptool: generated fsid 6b8c5e6b-dc21-4b86-85c9-629eae1ef431 2023-12-15T22:55:56.447 INFO:teuthology.orchestra.run.smithi071.stdout:setting min_mon_release = pacific 2023-12-15T22:55:56.447 INFO:teuthology.orchestra.run.smithi071.stdout:epoch 0 2023-12-15T22:55:56.447 INFO:teuthology.orchestra.run.smithi071.stdout:fsid 6b8c5e6b-dc21-4b86-85c9-629eae1ef431 2023-12-15T22:55:56.447 INFO:teuthology.orchestra.run.smithi071.stdout:last_changed 2023-12-15T22:55:56.447607+0000 2023-12-15T22:55:56.447 INFO:teuthology.orchestra.run.smithi071.stdout:created 2023-12-15T22:55:56.447607+0000 2023-12-15T22:55:56.447 INFO:teuthology.orchestra.run.smithi071.stdout:min_mon_release 16 (pacific) 2023-12-15T22:55:56.447 INFO:teuthology.orchestra.run.smithi071.stdout:election_strategy: 1 2023-12-15T22:55:56.447 INFO:teuthology.orchestra.run.smithi071.stdout:0: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.a 2023-12-15T22:55:56.447 INFO:teuthology.orchestra.run.smithi071.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-12-15T22:55:56.448 DEBUG:teuthology.orchestra.run.smithi071:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-15T22:55:56.462 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 6b8c5e6b-dc21-4b86-85c9-629eae1ef431... 2023-12-15T22:55:56.463 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-15T22:55:56.505 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-15T22:55:56.553 INFO:teuthology.orchestra.run.smithi071.stdout:[global] 2023-12-15T22:55:56.553 INFO:teuthology.orchestra.run.smithi071.stdout: chdir = "" 2023-12-15T22:55:56.553 INFO:teuthology.orchestra.run.smithi071.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-15T22:55:56.553 INFO:teuthology.orchestra.run.smithi071.stdout: auth supported = cephx 2023-12-15T22:55:56.553 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:56.553 INFO:teuthology.orchestra.run.smithi071.stdout: filestore xattr use omap = true 2023-12-15T22:55:56.553 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:56.553 INFO:teuthology.orchestra.run.smithi071.stdout: mon clock drift allowed = 1.000 2023-12-15T22:55:56.553 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:56.553 INFO:teuthology.orchestra.run.smithi071.stdout: osd crush chooseleaf type = 0 2023-12-15T22:55:56.553 INFO:teuthology.orchestra.run.smithi071.stdout: auth debug = true 2023-12-15T22:55:56.553 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:56.553 INFO:teuthology.orchestra.run.smithi071.stdout: ms die on old message = true 2023-12-15T22:55:56.553 INFO:teuthology.orchestra.run.smithi071.stdout: ms die on bug = true 2023-12-15T22:55:56.554 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:56.554 INFO:teuthology.orchestra.run.smithi071.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-15T22:55:56.554 INFO:teuthology.orchestra.run.smithi071.stdout: mon pg warn max object skew = 0 2023-12-15T22:55:56.554 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:56.554 INFO:teuthology.orchestra.run.smithi071.stdout: # disable pg_autoscaler by default for new pools 2023-12-15T22:55:56.554 INFO:teuthology.orchestra.run.smithi071.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-15T22:55:56.554 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:56.554 INFO:teuthology.orchestra.run.smithi071.stdout: osd pool default size = 2 2023-12-15T22:55:56.554 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:56.554 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd allow primary affinity = true 2023-12-15T22:55:56.554 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd allow pg remap = true 2023-12-15T22:55:56.554 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on legacy crush tunables = false 2023-12-15T22:55:56.554 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on crush straw calc version zero = false 2023-12-15T22:55:56.554 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on no sortbitwise = false 2023-12-15T22:55:56.554 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on osd down out interval zero = false 2023-12-15T22:55:56.554 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on too few osds = false 2023-12-15T22:55:56.555 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-15T22:55:56.555 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-15T22:55:56.555 INFO:teuthology.orchestra.run.smithi071.stdout: mon_allow_pool_size_one = true 2023-12-15T22:55:56.555 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:56.555 INFO:teuthology.orchestra.run.smithi071.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-15T22:55:56.555 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:56.555 INFO:teuthology.orchestra.run.smithi071.stdout: osd default data pool replay window = 5 2023-12-15T22:55:56.555 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:56.555 INFO:teuthology.orchestra.run.smithi071.stdout: mon allow pool delete = true 2023-12-15T22:55:56.555 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:56.555 INFO:teuthology.orchestra.run.smithi071.stdout: mon cluster log file level = debug 2023-12-15T22:55:56.555 INFO:teuthology.orchestra.run.smithi071.stdout: debug asserts on shutdown = true 2023-12-15T22:55:56.555 INFO:teuthology.orchestra.run.smithi071.stdout: mon health detail to clog = false 2023-12-15T22:55:56.555 INFO:teuthology.orchestra.run.smithi071.stdout: mon host = 172.21.15.71 2023-12-15T22:55:56.555 INFO:teuthology.orchestra.run.smithi071.stdout: enable experimental unrecoverable data corrupting features = crimson 2023-12-15T22:55:56.556 INFO:teuthology.orchestra.run.smithi071.stdout: ms client mode = crc 2023-12-15T22:55:56.556 INFO:teuthology.orchestra.run.smithi071.stdout: ms cluster mode = crc 2023-12-15T22:55:56.556 INFO:teuthology.orchestra.run.smithi071.stdout: ms mon client mode = crc 2023-12-15T22:55:56.556 INFO:teuthology.orchestra.run.smithi071.stdout: ms mon cluster mode = crc 2023-12-15T22:55:56.556 INFO:teuthology.orchestra.run.smithi071.stdout: ms mon service mode = crc 2023-12-15T22:55:56.556 INFO:teuthology.orchestra.run.smithi071.stdout: ms service mode = crc 2023-12-15T22:55:56.556 INFO:teuthology.orchestra.run.smithi071.stdout: osd_pool_default_min_size = 2 2023-12-15T22:55:56.556 INFO:teuthology.orchestra.run.smithi071.stdout: osd_pool_default_size = 3 2023-12-15T22:55:56.556 INFO:teuthology.orchestra.run.smithi071.stdout: fsid = 6b8c5e6b-dc21-4b86-85c9-629eae1ef431 2023-12-15T22:55:56.556 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:56.556 INFO:teuthology.orchestra.run.smithi071.stdout:[osd] 2023-12-15T22:55:56.556 INFO:teuthology.orchestra.run.smithi071.stdout: osd journal size = 100 2023-12-15T22:55:56.556 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:56.556 INFO:teuthology.orchestra.run.smithi071.stdout: osd scrub load threshold = 5.0 2023-12-15T22:55:56.556 INFO:teuthology.orchestra.run.smithi071.stdout: osd scrub max interval = 120 2023-12-15T22:55:56.556 INFO:teuthology.orchestra.run.smithi071.stdout: osd mclock profile = high_recovery_ops 2023-12-15T22:55:56.557 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:56.557 INFO:teuthology.orchestra.run.smithi071.stdout: osd recover clone overlap = true 2023-12-15T22:55:56.557 INFO:teuthology.orchestra.run.smithi071.stdout: osd recovery max chunk = 1048576 2023-12-15T22:55:56.557 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:56.557 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug shutdown = true 2023-12-15T22:55:56.557 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug op order = true 2023-12-15T22:55:56.557 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug verify stray on activate = true 2023-12-15T22:55:56.557 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug trim objects = true 2023-12-15T22:55:56.557 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:56.557 INFO:teuthology.orchestra.run.smithi071.stdout: osd open classes on start = true 2023-12-15T22:55:56.557 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug pg log writeout = true 2023-12-15T22:55:56.557 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:56.557 INFO:teuthology.orchestra.run.smithi071.stdout: osd deep scrub update digest min age = 30 2023-12-15T22:55:56.557 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:56.557 INFO:teuthology.orchestra.run.smithi071.stdout: osd map max advance = 10 2023-12-15T22:55:56.557 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:56.558 INFO:teuthology.orchestra.run.smithi071.stdout: journal zero on create = true 2023-12-15T22:55:56.558 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:56.558 INFO:teuthology.orchestra.run.smithi071.stdout: filestore ondisk finisher threads = 3 2023-12-15T22:55:56.558 INFO:teuthology.orchestra.run.smithi071.stdout: filestore apply finisher threads = 3 2023-12-15T22:55:56.558 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:56.558 INFO:teuthology.orchestra.run.smithi071.stdout: bdev debug aio = true 2023-12-15T22:55:56.558 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug misdirected ops = true 2023-12-15T22:55:56.558 INFO:teuthology.orchestra.run.smithi071.stdout: debug monc = 20 2023-12-15T22:55:56.558 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore block size = 96636764160 2023-12-15T22:55:56.558 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore compression algorithm = snappy 2023-12-15T22:55:56.558 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore compression mode = aggressive 2023-12-15T22:55:56.558 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore fsck on mount = True 2023-12-15T22:55:56.558 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore rocksdb cf = False 2023-12-15T22:55:56.558 INFO:teuthology.orchestra.run.smithi071.stdout: crimson seastar smp = 3 2023-12-15T22:55:56.558 INFO:teuthology.orchestra.run.smithi071.stdout: debug bluefs = 20 2023-12-15T22:55:56.558 INFO:teuthology.orchestra.run.smithi071.stdout: debug bluestore = 20 2023-12-15T22:55:56.558 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2023-12-15T22:55:56.559 INFO:teuthology.orchestra.run.smithi071.stdout: debug osd = 20 2023-12-15T22:55:56.559 INFO:teuthology.orchestra.run.smithi071.stdout: debug rocksdb = 10 2023-12-15T22:55:56.559 INFO:teuthology.orchestra.run.smithi071.stdout: err to stderr = True 2023-12-15T22:55:56.559 INFO:teuthology.orchestra.run.smithi071.stdout: log flush on exit = True 2023-12-15T22:55:56.559 INFO:teuthology.orchestra.run.smithi071.stdout: log to file = False 2023-12-15T22:55:56.559 INFO:teuthology.orchestra.run.smithi071.stdout: log to stderr = True 2023-12-15T22:55:56.559 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd backfillfull_ratio = 0.85 2023-12-15T22:55:56.559 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd full ratio = 0.9 2023-12-15T22:55:56.559 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd nearfull ratio = 0.8 2023-12-15T22:55:56.559 INFO:teuthology.orchestra.run.smithi071.stdout: osd blocked scrub grace period = 3600 2023-12-15T22:55:56.559 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug reject backfill probability = 0.3 2023-12-15T22:55:56.559 INFO:teuthology.orchestra.run.smithi071.stdout: osd delete sleep = 1 2023-12-15T22:55:56.559 INFO:teuthology.orchestra.run.smithi071.stdout: osd failsafe full ratio = 0.95 2023-12-15T22:55:56.559 INFO:teuthology.orchestra.run.smithi071.stdout: osd max backfills = 3 2023-12-15T22:55:56.559 INFO:teuthology.orchestra.run.smithi071.stdout: osd max markdown count = 1000 2023-12-15T22:55:56.560 INFO:teuthology.orchestra.run.smithi071.stdout: osd objectstore = bluestore 2023-12-15T22:55:56.560 INFO:teuthology.orchestra.run.smithi071.stdout: osd scrub min interval = 60 2023-12-15T22:55:56.560 INFO:teuthology.orchestra.run.smithi071.stdout: osd shutdown pgref assert = True 2023-12-15T22:55:56.560 INFO:teuthology.orchestra.run.smithi071.stdout: osd snap trim sleep = 2 2023-12-15T22:55:56.560 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:56.560 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr] 2023-12-15T22:55:56.560 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2023-12-15T22:55:56.560 INFO:teuthology.orchestra.run.smithi071.stdout: debug mgr = 20 2023-12-15T22:55:56.560 INFO:teuthology.orchestra.run.smithi071.stdout: debug mon = 20 2023-12-15T22:55:56.560 INFO:teuthology.orchestra.run.smithi071.stdout: debug auth = 20 2023-12-15T22:55:56.560 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min pgs per osd = 4 2023-12-15T22:55:56.560 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min bytes per osd = 10 2023-12-15T22:55:56.560 INFO:teuthology.orchestra.run.smithi071.stdout: mgr/telemetry/nag = false 2023-12-15T22:55:56.560 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:56.560 INFO:teuthology.orchestra.run.smithi071.stdout:[mon] 2023-12-15T22:55:56.560 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2023-12-15T22:55:56.561 INFO:teuthology.orchestra.run.smithi071.stdout: debug mon = 20 2023-12-15T22:55:56.561 INFO:teuthology.orchestra.run.smithi071.stdout: debug paxos = 20 2023-12-15T22:55:56.561 INFO:teuthology.orchestra.run.smithi071.stdout: debug auth = 20 2023-12-15T22:55:56.561 INFO:teuthology.orchestra.run.smithi071.stdout: mon data avail warn = 5 2023-12-15T22:55:56.561 INFO:teuthology.orchestra.run.smithi071.stdout: mon mgr mkfs grace = 240 2023-12-15T22:55:56.561 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min pgs per osd = 4 2023-12-15T22:55:56.561 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd reporter subtree level = osd 2023-12-15T22:55:56.561 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd prime pg temp = true 2023-12-15T22:55:56.561 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min bytes per osd = 10 2023-12-15T22:55:56.561 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:56.561 INFO:teuthology.orchestra.run.smithi071.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-15T22:55:56.561 INFO:teuthology.orchestra.run.smithi071.stdout: auth mon ticket ttl = 660 # 11m 2023-12-15T22:55:56.561 INFO:teuthology.orchestra.run.smithi071.stdout: auth service ticket ttl = 240 # 4m 2023-12-15T22:55:56.561 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:56.561 INFO:teuthology.orchestra.run.smithi071.stdout: # don't complain about insecure global_id in the test suite 2023-12-15T22:55:56.561 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-15T22:55:56.562 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-15T22:55:56.562 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:56.562 INFO:teuthology.orchestra.run.smithi071.stdout: # 1m isn't quite enough 2023-12-15T22:55:56.562 INFO:teuthology.orchestra.run.smithi071.stdout: mon_down_mkfs_grace = 2m 2023-12-15T22:55:56.562 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:56.562 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_filestore_osds = false 2023-12-15T22:55:56.562 INFO:teuthology.orchestra.run.smithi071.stdout: mon min osdmap epochs = 50 2023-12-15T22:55:56.562 INFO:teuthology.orchestra.run.smithi071.stdout: mon osdmap full prune interval = 2 2023-12-15T22:55:56.562 INFO:teuthology.orchestra.run.smithi071.stdout: mon osdmap full prune min = 15 2023-12-15T22:55:56.562 INFO:teuthology.orchestra.run.smithi071.stdout: mon osdmap full prune txsize = 2 2023-12-15T22:55:56.562 INFO:teuthology.orchestra.run.smithi071.stdout: osd pool default crimson = True 2023-12-15T22:55:56.562 INFO:teuthology.orchestra.run.smithi071.stdout: paxos service trim min = 10 2023-12-15T22:55:56.562 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:55:56.562 INFO:teuthology.orchestra.run.smithi071.stdout:[client] 2023-12-15T22:55:56.562 INFO:teuthology.orchestra.run.smithi071.stdout: rgw cache enabled = true 2023-12-15T22:55:56.562 INFO:teuthology.orchestra.run.smithi071.stdout: rgw enable ops log = true 2023-12-15T22:55:56.563 INFO:teuthology.orchestra.run.smithi071.stdout: rgw enable usage log = true 2023-12-15T22:55:56.563 INFO:teuthology.orchestra.run.smithi071.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-15T22:55:56.563 INFO:teuthology.orchestra.run.smithi071.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-15T22:55:56.563 INFO:teuthology.orchestra.run.smithi071.stdout:[mon.a] 2023-12-15T22:55:56.563 INFO:teuthology.orchestra.run.smithi072.stdout:[global] 2023-12-15T22:55:56.563 INFO:teuthology.orchestra.run.smithi072.stdout: chdir = "" 2023-12-15T22:55:56.563 INFO:teuthology.orchestra.run.smithi072.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-15T22:55:56.563 INFO:teuthology.orchestra.run.smithi072.stdout: auth supported = cephx 2023-12-15T22:55:56.563 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:56.563 INFO:teuthology.orchestra.run.smithi072.stdout: filestore xattr use omap = true 2023-12-15T22:55:56.563 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:56.563 INFO:teuthology.orchestra.run.smithi072.stdout: mon clock drift allowed = 1.000 2023-12-15T22:55:56.564 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:56.564 INFO:teuthology.orchestra.run.smithi072.stdout: osd crush chooseleaf type = 0 2023-12-15T22:55:56.564 INFO:teuthology.orchestra.run.smithi072.stdout: auth debug = true 2023-12-15T22:55:56.564 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:56.564 INFO:teuthology.orchestra.run.smithi072.stdout: ms die on old message = true 2023-12-15T22:55:56.564 INFO:teuthology.orchestra.run.smithi072.stdout: ms die on bug = true 2023-12-15T22:55:56.564 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:56.564 INFO:teuthology.orchestra.run.smithi072.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-15T22:55:56.564 INFO:teuthology.orchestra.run.smithi072.stdout: mon pg warn max object skew = 0 2023-12-15T22:55:56.564 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:56.564 INFO:teuthology.orchestra.run.smithi072.stdout: # disable pg_autoscaler by default for new pools 2023-12-15T22:55:56.564 INFO:teuthology.orchestra.run.smithi072.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-15T22:55:56.564 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:56.564 INFO:teuthology.orchestra.run.smithi072.stdout: osd pool default size = 2 2023-12-15T22:55:56.564 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:56.564 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd allow primary affinity = true 2023-12-15T22:55:56.564 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd allow pg remap = true 2023-12-15T22:55:56.565 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on legacy crush tunables = false 2023-12-15T22:55:56.565 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on crush straw calc version zero = false 2023-12-15T22:55:56.565 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on no sortbitwise = false 2023-12-15T22:55:56.565 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on osd down out interval zero = false 2023-12-15T22:55:56.565 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on too few osds = false 2023-12-15T22:55:56.565 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-15T22:55:56.565 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-15T22:55:56.565 INFO:teuthology.orchestra.run.smithi072.stdout: mon_allow_pool_size_one = true 2023-12-15T22:55:56.565 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:56.565 INFO:teuthology.orchestra.run.smithi072.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-15T22:55:56.565 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:56.565 INFO:teuthology.orchestra.run.smithi072.stdout: osd default data pool replay window = 5 2023-12-15T22:55:56.565 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:56.565 INFO:teuthology.orchestra.run.smithi072.stdout: mon allow pool delete = true 2023-12-15T22:55:56.566 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:56.566 INFO:teuthology.orchestra.run.smithi072.stdout: mon cluster log file level = debug 2023-12-15T22:55:56.566 INFO:teuthology.orchestra.run.smithi072.stdout: debug asserts on shutdown = true 2023-12-15T22:55:56.566 INFO:teuthology.orchestra.run.smithi072.stdout: mon health detail to clog = false 2023-12-15T22:55:56.566 INFO:teuthology.orchestra.run.smithi072.stdout: mon host = 172.21.15.71 2023-12-15T22:55:56.566 INFO:teuthology.orchestra.run.smithi072.stdout: enable experimental unrecoverable data corrupting features = crimson 2023-12-15T22:55:56.566 INFO:teuthology.orchestra.run.smithi072.stdout: ms client mode = crc 2023-12-15T22:55:56.566 INFO:teuthology.orchestra.run.smithi072.stdout: ms cluster mode = crc 2023-12-15T22:55:56.566 INFO:teuthology.orchestra.run.smithi072.stdout: ms mon client mode = crc 2023-12-15T22:55:56.566 INFO:teuthology.orchestra.run.smithi072.stdout: ms mon cluster mode = crc 2023-12-15T22:55:56.566 INFO:teuthology.orchestra.run.smithi072.stdout: ms mon service mode = crc 2023-12-15T22:55:56.566 INFO:teuthology.orchestra.run.smithi072.stdout: ms service mode = crc 2023-12-15T22:55:56.566 INFO:teuthology.orchestra.run.smithi072.stdout: osd_pool_default_min_size = 2 2023-12-15T22:55:56.566 INFO:teuthology.orchestra.run.smithi072.stdout: osd_pool_default_size = 3 2023-12-15T22:55:56.566 INFO:teuthology.orchestra.run.smithi072.stdout: fsid = 6b8c5e6b-dc21-4b86-85c9-629eae1ef431 2023-12-15T22:55:56.566 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:56.566 INFO:teuthology.orchestra.run.smithi072.stdout:[osd] 2023-12-15T22:55:56.567 INFO:teuthology.orchestra.run.smithi072.stdout: osd journal size = 100 2023-12-15T22:55:56.567 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:56.567 INFO:teuthology.orchestra.run.smithi072.stdout: osd scrub load threshold = 5.0 2023-12-15T22:55:56.567 INFO:teuthology.orchestra.run.smithi072.stdout: osd scrub max interval = 120 2023-12-15T22:55:56.567 INFO:teuthology.orchestra.run.smithi072.stdout: osd mclock profile = high_recovery_ops 2023-12-15T22:55:56.567 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:56.567 INFO:teuthology.orchestra.run.smithi072.stdout: osd recover clone overlap = true 2023-12-15T22:55:56.567 INFO:teuthology.orchestra.run.smithi072.stdout: osd recovery max chunk = 1048576 2023-12-15T22:55:56.567 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:56.567 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug shutdown = true 2023-12-15T22:55:56.567 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug op order = true 2023-12-15T22:55:56.567 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug verify stray on activate = true 2023-12-15T22:55:56.567 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug trim objects = true 2023-12-15T22:55:56.567 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:56.567 INFO:teuthology.orchestra.run.smithi072.stdout: osd open classes on start = true 2023-12-15T22:55:56.567 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug pg log writeout = true 2023-12-15T22:55:56.567 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:56.568 INFO:teuthology.orchestra.run.smithi072.stdout: osd deep scrub update digest min age = 30 2023-12-15T22:55:56.568 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:56.568 INFO:teuthology.orchestra.run.smithi072.stdout: osd map max advance = 10 2023-12-15T22:55:56.568 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:56.568 INFO:teuthology.orchestra.run.smithi072.stdout: journal zero on create = true 2023-12-15T22:55:56.568 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:56.568 INFO:teuthology.orchestra.run.smithi072.stdout: filestore ondisk finisher threads = 3 2023-12-15T22:55:56.568 INFO:teuthology.orchestra.run.smithi072.stdout: filestore apply finisher threads = 3 2023-12-15T22:55:56.568 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:56.568 INFO:teuthology.orchestra.run.smithi072.stdout: bdev debug aio = true 2023-12-15T22:55:56.568 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug misdirected ops = true 2023-12-15T22:55:56.568 INFO:teuthology.orchestra.run.smithi072.stdout: debug monc = 20 2023-12-15T22:55:56.568 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore block size = 96636764160 2023-12-15T22:55:56.568 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore compression algorithm = snappy 2023-12-15T22:55:56.568 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore compression mode = aggressive 2023-12-15T22:55:56.568 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore fsck on mount = True 2023-12-15T22:55:56.569 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore rocksdb cf = False 2023-12-15T22:55:56.569 INFO:teuthology.orchestra.run.smithi072.stdout: crimson seastar smp = 3 2023-12-15T22:55:56.569 INFO:teuthology.orchestra.run.smithi072.stdout: debug bluefs = 20 2023-12-15T22:55:56.569 INFO:teuthology.orchestra.run.smithi072.stdout: debug bluestore = 20 2023-12-15T22:55:56.569 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2023-12-15T22:55:56.569 INFO:teuthology.orchestra.run.smithi072.stdout: debug osd = 20 2023-12-15T22:55:56.569 INFO:teuthology.orchestra.run.smithi072.stdout: debug rocksdb = 10 2023-12-15T22:55:56.569 INFO:teuthology.orchestra.run.smithi072.stdout: err to stderr = True 2023-12-15T22:55:56.569 INFO:teuthology.orchestra.run.smithi072.stdout: log flush on exit = True 2023-12-15T22:55:56.569 INFO:teuthology.orchestra.run.smithi072.stdout: log to file = False 2023-12-15T22:55:56.569 INFO:teuthology.orchestra.run.smithi072.stdout: log to stderr = True 2023-12-15T22:55:56.569 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd backfillfull_ratio = 0.85 2023-12-15T22:55:56.569 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd full ratio = 0.9 2023-12-15T22:55:56.569 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd nearfull ratio = 0.8 2023-12-15T22:55:56.569 INFO:teuthology.orchestra.run.smithi072.stdout: osd blocked scrub grace period = 3600 2023-12-15T22:55:56.569 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug reject backfill probability = 0.3 2023-12-15T22:55:56.570 INFO:teuthology.orchestra.run.smithi072.stdout: osd delete sleep = 1 2023-12-15T22:55:56.570 INFO:teuthology.orchestra.run.smithi072.stdout: osd failsafe full ratio = 0.95 2023-12-15T22:55:56.570 INFO:teuthology.orchestra.run.smithi072.stdout: osd max backfills = 3 2023-12-15T22:55:56.570 INFO:teuthology.orchestra.run.smithi072.stdout: osd max markdown count = 1000 2023-12-15T22:55:56.570 INFO:teuthology.orchestra.run.smithi072.stdout: osd objectstore = bluestore 2023-12-15T22:55:56.570 INFO:teuthology.orchestra.run.smithi072.stdout: osd scrub min interval = 60 2023-12-15T22:55:56.570 INFO:teuthology.orchestra.run.smithi072.stdout: osd shutdown pgref assert = True 2023-12-15T22:55:56.570 INFO:teuthology.orchestra.run.smithi072.stdout: osd snap trim sleep = 2 2023-12-15T22:55:56.570 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:56.570 INFO:teuthology.orchestra.run.smithi072.stdout:[mgr] 2023-12-15T22:55:56.570 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2023-12-15T22:55:56.570 INFO:teuthology.orchestra.run.smithi072.stdout: debug mgr = 20 2023-12-15T22:55:56.570 INFO:teuthology.orchestra.run.smithi072.stdout: debug mon = 20 2023-12-15T22:55:56.570 INFO:teuthology.orchestra.run.smithi072.stdout: debug auth = 20 2023-12-15T22:55:56.570 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min pgs per osd = 4 2023-12-15T22:55:56.570 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min bytes per osd = 10 2023-12-15T22:55:56.571 INFO:teuthology.orchestra.run.smithi072.stdout: mgr/telemetry/nag = false 2023-12-15T22:55:56.571 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:56.571 INFO:teuthology.orchestra.run.smithi072.stdout:[mon] 2023-12-15T22:55:56.571 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2023-12-15T22:55:56.571 INFO:teuthology.orchestra.run.smithi072.stdout: debug mon = 20 2023-12-15T22:55:56.571 INFO:teuthology.orchestra.run.smithi072.stdout: debug paxos = 20 2023-12-15T22:55:56.571 INFO:teuthology.orchestra.run.smithi072.stdout: debug auth = 20 2023-12-15T22:55:56.571 INFO:teuthology.orchestra.run.smithi072.stdout: mon data avail warn = 5 2023-12-15T22:55:56.571 INFO:teuthology.orchestra.run.smithi072.stdout: mon mgr mkfs grace = 240 2023-12-15T22:55:56.571 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min pgs per osd = 4 2023-12-15T22:55:56.571 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd reporter subtree level = osd 2023-12-15T22:55:56.571 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd prime pg temp = true 2023-12-15T22:55:56.571 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min bytes per osd = 10 2023-12-15T22:55:56.571 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:56.571 INFO:teuthology.orchestra.run.smithi072.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-15T22:55:56.571 INFO:teuthology.orchestra.run.smithi072.stdout: auth mon ticket ttl = 660 # 11m 2023-12-15T22:55:56.572 INFO:teuthology.orchestra.run.smithi072.stdout: auth service ticket ttl = 240 # 4m 2023-12-15T22:55:56.572 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:56.572 INFO:teuthology.orchestra.run.smithi072.stdout: # don't complain about insecure global_id in the test suite 2023-12-15T22:55:56.572 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-15T22:55:56.572 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-15T22:55:56.572 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:56.572 INFO:teuthology.orchestra.run.smithi072.stdout: # 1m isn't quite enough 2023-12-15T22:55:56.572 INFO:teuthology.orchestra.run.smithi072.stdout: mon_down_mkfs_grace = 2m 2023-12-15T22:55:56.572 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:56.572 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_filestore_osds = false 2023-12-15T22:55:56.572 INFO:teuthology.orchestra.run.smithi072.stdout: mon min osdmap epochs = 50 2023-12-15T22:55:56.572 INFO:teuthology.orchestra.run.smithi072.stdout: mon osdmap full prune interval = 2 2023-12-15T22:55:56.572 INFO:teuthology.orchestra.run.smithi072.stdout: mon osdmap full prune min = 15 2023-12-15T22:55:56.572 INFO:teuthology.orchestra.run.smithi072.stdout: mon osdmap full prune txsize = 2 2023-12-15T22:55:56.572 INFO:teuthology.orchestra.run.smithi072.stdout: osd pool default crimson = True 2023-12-15T22:55:56.572 INFO:teuthology.orchestra.run.smithi072.stdout: paxos service trim min = 10 2023-12-15T22:55:56.572 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-12-15T22:55:56.572 INFO:teuthology.orchestra.run.smithi072.stdout:[client] 2023-12-15T22:55:56.573 INFO:teuthology.orchestra.run.smithi072.stdout: rgw cache enabled = true 2023-12-15T22:55:56.573 INFO:teuthology.orchestra.run.smithi072.stdout: rgw enable ops log = true 2023-12-15T22:55:56.573 INFO:teuthology.orchestra.run.smithi072.stdout: rgw enable usage log = true 2023-12-15T22:55:56.573 INFO:teuthology.orchestra.run.smithi072.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-15T22:55:56.573 INFO:teuthology.orchestra.run.smithi072.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-15T22:55:56.573 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.a] 2023-12-15T22:55:56.574 INFO:tasks.ceph:Creating admin key on mon.a... 2023-12-15T22:55:56.574 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-12-15T22:55:56.636 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-15T22:55:56.636 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-12-15T22:55:56.636 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-12-15T22:55:56.650 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-12-15T22:55:56.651 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-12-15T22:55:56.705 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi071.front.sepia.ceph.com 2023-12-15T22:55:56.705 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-12-15T22:55:56.705 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-15T22:55:56.705 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-15T22:55:56.783 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-12-15T22:55:56.783 DEBUG:teuthology.orchestra.run.smithi071:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-15T22:55:56.837 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi072.front.sepia.ceph.com 2023-12-15T22:55:56.838 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2023-12-15T22:55:56.838 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-15T22:55:56.838 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-15T22:55:56.876 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2023-12-15T22:55:56.877 DEBUG:teuthology.orchestra.run.smithi072:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-15T22:55:56.932 INFO:tasks.ceph:Setting up mon nodes... 2023-12-15T22:55:56.932 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-15T22:55:56.932 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-12-15T22:55:57.039 INFO:teuthology.orchestra.run.smithi072.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-12-15T22:55:57.041 INFO:tasks.ceph:Setting up mds nodes... 2023-12-15T22:55:57.042 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-15T22:55:57.042 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-12-15T22:55:57.090 INFO:teuthology.orchestra.run.smithi071.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-12-15T22:55:57.106 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-12-15T22:55:57.154 INFO:teuthology.orchestra.run.smithi072.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-12-15T22:55:57.170 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-15T22:55:57.170 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi071.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi072.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2023-12-15T22:55:57.170 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-12-15T22:55:57.196 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2023-12-15T22:55:57.196 INFO:tasks.ceph:role: osd.0 2023-12-15T22:55:57.196 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi071.front.sepia.ceph.com 2023-12-15T22:55:57.197 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-15T22:55:57.271 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-15T22:55:57.271 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-15T22:55:57.271 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-15T22:55:57.271 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-15T22:55:57.271 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-15T22:55:57.271 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2023-12-15T22:55:57.271 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-15T22:55:57.271 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-15T22:55:57.271 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-15T22:55:57.271 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-15T22:55:57.777 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2023-12-15T22:55:57.778 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2023-12-15T22:55:57.779 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-12-15T22:55:57.877 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-12-15T22:55:57.914 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-12-15T22:55:57.981 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2023-12-15T22:55:57.981 INFO:tasks.ceph:role: osd.1 2023-12-15T22:55:57.982 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi071.front.sepia.ceph.com 2023-12-15T22:55:57.982 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-15T22:55:58.057 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-15T22:55:58.057 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-15T22:55:58.057 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-15T22:55:58.057 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-15T22:55:58.057 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-15T22:55:58.057 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2023-12-15T22:55:58.057 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-15T22:55:58.057 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-15T22:55:58.057 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-15T22:55:58.058 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-15T22:55:58.523 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2023-12-15T22:55:58.525 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2023-12-15T22:55:58.526 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-12-15T22:55:58.620 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-12-15T22:55:58.650 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-15T22:55:59.169 INFO:teuthology.orchestra.run.smithi071.stderr:WARNING: debug mode. Not for benchmarking or production 2023-12-15T22:55:59.169 INFO:teuthology.orchestra.run.smithi071.stderr:INFO 2023-12-15 22:55:59,170 seastar - Reactor backend: linux-aio 2023-12-15T22:55:59.185 INFO:teuthology.orchestra.run.smithi071.stderr:INFO 2023-12-15 22:55:59,186 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-12-15T22:55:59.185 INFO:teuthology.orchestra.run.smithi071.stderr:INFO 2023-12-15 22:55:59,186 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-12-15T22:55:59.186 INFO:teuthology.orchestra.run.smithi071.stderr:INFO 2023-12-15 22:55:59,186 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-12-15T22:55:59.186 INFO:teuthology.orchestra.run.smithi071.stderr:INFO 2023-12-15 22:55:59,186 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-12-15T22:55:59.187 INFO:teuthology.orchestra.run.smithi071.stderr:==80703==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-12-15T22:55:59.193 INFO:teuthology.orchestra.run.smithi071.stderr:==80703==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffef940b000; bottom 0x7f4f3c381000; size: 0x00afbd08a000 (754790735872) 2023-12-15T22:55:59.193 INFO:teuthology.orchestra.run.smithi071.stderr:False positive error reports may follow 2023-12-15T22:55:59.193 INFO:teuthology.orchestra.run.smithi071.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-12-15T22:55:59.229 INFO:teuthology.orchestra.run.smithi071.stderr:INFO 2023-12-15 22:55:59,230 [shard 0] osd - bypassing the config fetch due to --no-mon-config 2023-12-15T22:55:59.229 INFO:teuthology.orchestra.run.smithi071.stderr:INFO 2023-12-15 22:55:59,230 [shard 0] osd - get_early_config: set --smp 3 2023-12-15T22:55:59.715 INFO:teuthology.orchestra.run.smithi071.stderr:----------------------------------------------------- 2023-12-15T22:55:59.715 INFO:teuthology.orchestra.run.smithi071.stderr:Suppressions used: 2023-12-15T22:55:59.715 INFO:teuthology.orchestra.run.smithi071.stderr: count bytes template 2023-12-15T22:55:59.715 INFO:teuthology.orchestra.run.smithi071.stderr: 1 8 MallocExtension::Register 2023-12-15T22:55:59.715 INFO:teuthology.orchestra.run.smithi071.stderr:----------------------------------------------------- 2023-12-15T22:55:59.715 INFO:teuthology.orchestra.run.smithi071.stderr: 2023-12-15T22:55:59.765 INFO:teuthology.orchestra.run.smithi071.stderr:WARNING: debug mode. Not for benchmarking or production 2023-12-15T22:55:59.766 INFO:teuthology.orchestra.run.smithi071.stderr:INFO 2023-12-15 22:55:59,767 seastar - Reactor backend: linux-aio 2023-12-15T22:55:59.784 INFO:teuthology.orchestra.run.smithi071.stderr:INFO 2023-12-15 22:55:59,784 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-12-15T22:55:59.784 INFO:teuthology.orchestra.run.smithi071.stderr:INFO 2023-12-15 22:55:59,785 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-12-15T22:55:59.784 INFO:teuthology.orchestra.run.smithi071.stderr:INFO 2023-12-15 22:55:59,785 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-12-15T22:55:59.791 INFO:teuthology.orchestra.run.smithi071.stderr:INFO 2023-12-15 22:55:59,792 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-12-15T22:55:59.794 INFO:teuthology.orchestra.run.smithi071.stderr:==80701==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-12-15T22:55:59.799 INFO:teuthology.orchestra.run.smithi071.stderr:==80701==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffef940b000; bottom 0x7f4f3a16a000; size: 0x00afbf2a1000 (754826481664) 2023-12-15T22:55:59.800 INFO:teuthology.orchestra.run.smithi071.stderr:False positive error reports may follow 2023-12-15T22:55:59.800 INFO:teuthology.orchestra.run.smithi071.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-12-15T22:56:03.800 INFO:teuthology.orchestra.run.smithi071.stderr:----------------------------------------------------- 2023-12-15T22:56:03.800 INFO:teuthology.orchestra.run.smithi071.stderr:Suppressions used: 2023-12-15T22:56:03.800 INFO:teuthology.orchestra.run.smithi071.stderr: count bytes template 2023-12-15T22:56:03.801 INFO:teuthology.orchestra.run.smithi071.stderr: 1 8 MallocExtension::Register 2023-12-15T22:56:03.801 INFO:teuthology.orchestra.run.smithi071.stderr:----------------------------------------------------- 2023-12-15T22:56:03.801 INFO:teuthology.orchestra.run.smithi071.stderr: 2023-12-15T22:56:03.802 INFO:teuthology.orchestra.run.smithi071.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 6b8c5e6b-dc21-4b86-85c9-629eae1ef431 2023-12-15T22:56:03.836 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-12-15T22:56:03.864 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-15T22:56:04.367 INFO:teuthology.orchestra.run.smithi071.stderr:WARNING: debug mode. Not for benchmarking or production 2023-12-15T22:56:04.368 INFO:teuthology.orchestra.run.smithi071.stderr:INFO 2023-12-15 22:56:04,368 seastar - Reactor backend: linux-aio 2023-12-15T22:56:04.383 INFO:teuthology.orchestra.run.smithi071.stderr:INFO 2023-12-15 22:56:04,384 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-12-15T22:56:04.384 INFO:teuthology.orchestra.run.smithi071.stderr:INFO 2023-12-15 22:56:04,384 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-12-15T22:56:04.384 INFO:teuthology.orchestra.run.smithi071.stderr:INFO 2023-12-15 22:56:04,384 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-12-15T22:56:04.384 INFO:teuthology.orchestra.run.smithi071.stderr:INFO 2023-12-15 22:56:04,384 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-12-15T22:56:04.385 INFO:teuthology.orchestra.run.smithi071.stderr:==80885==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-12-15T22:56:04.391 INFO:teuthology.orchestra.run.smithi071.stderr:==80885==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffeeb12c000; bottom 0x7f26533c6000; size: 0x00d897d66000 (930260344832) 2023-12-15T22:56:04.392 INFO:teuthology.orchestra.run.smithi071.stderr:False positive error reports may follow 2023-12-15T22:56:04.392 INFO:teuthology.orchestra.run.smithi071.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-12-15T22:56:04.427 INFO:teuthology.orchestra.run.smithi071.stderr:INFO 2023-12-15 22:56:04,427 [shard 0] osd - bypassing the config fetch due to --no-mon-config 2023-12-15T22:56:04.427 INFO:teuthology.orchestra.run.smithi071.stderr:INFO 2023-12-15 22:56:04,428 [shard 0] osd - get_early_config: set --smp 3 2023-12-15T22:56:04.893 INFO:teuthology.orchestra.run.smithi071.stderr:----------------------------------------------------- 2023-12-15T22:56:04.893 INFO:teuthology.orchestra.run.smithi071.stderr:Suppressions used: 2023-12-15T22:56:04.893 INFO:teuthology.orchestra.run.smithi071.stderr: count bytes template 2023-12-15T22:56:04.894 INFO:teuthology.orchestra.run.smithi071.stderr: 1 8 MallocExtension::Register 2023-12-15T22:56:04.894 INFO:teuthology.orchestra.run.smithi071.stderr:----------------------------------------------------- 2023-12-15T22:56:04.894 INFO:teuthology.orchestra.run.smithi071.stderr: 2023-12-15T22:56:04.947 INFO:teuthology.orchestra.run.smithi071.stderr:WARNING: debug mode. Not for benchmarking or production 2023-12-15T22:56:04.948 INFO:teuthology.orchestra.run.smithi071.stderr:INFO 2023-12-15 22:56:04,949 seastar - Reactor backend: linux-aio 2023-12-15T22:56:04.966 INFO:teuthology.orchestra.run.smithi071.stderr:INFO 2023-12-15 22:56:04,967 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-12-15T22:56:04.966 INFO:teuthology.orchestra.run.smithi071.stderr:INFO 2023-12-15 22:56:04,967 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-12-15T22:56:04.967 INFO:teuthology.orchestra.run.smithi071.stderr:INFO 2023-12-15 22:56:04,967 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-12-15T22:56:04.973 INFO:teuthology.orchestra.run.smithi071.stderr:INFO 2023-12-15 22:56:04,974 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-12-15T22:56:04.976 INFO:teuthology.orchestra.run.smithi071.stderr:==80883==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-12-15T22:56:04.982 INFO:teuthology.orchestra.run.smithi071.stderr:==80883==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffeeb12c000; bottom 0x7f26511af000; size: 0x00d899f7d000 (930296090624) 2023-12-15T22:56:04.982 INFO:teuthology.orchestra.run.smithi071.stderr:False positive error reports may follow 2023-12-15T22:56:04.982 INFO:teuthology.orchestra.run.smithi071.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-12-15T22:56:08.988 INFO:teuthology.orchestra.run.smithi071.stderr:----------------------------------------------------- 2023-12-15T22:56:08.988 INFO:teuthology.orchestra.run.smithi071.stderr:Suppressions used: 2023-12-15T22:56:08.988 INFO:teuthology.orchestra.run.smithi071.stderr: count bytes template 2023-12-15T22:56:08.988 INFO:teuthology.orchestra.run.smithi071.stderr: 1 8 MallocExtension::Register 2023-12-15T22:56:08.988 INFO:teuthology.orchestra.run.smithi071.stderr:----------------------------------------------------- 2023-12-15T22:56:08.988 INFO:teuthology.orchestra.run.smithi071.stderr: 2023-12-15T22:56:08.988 INFO:teuthology.orchestra.run.smithi071.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 6b8c5e6b-dc21-4b86-85c9-629eae1ef431 2023-12-15T22:56:09.024 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-12-15T22:56:09.050 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-12-15T22:56:09.084 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2023-12-15T22:56:09.084 INFO:tasks.ceph:role: osd.2 2023-12-15T22:56:09.084 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi072.front.sepia.ceph.com 2023-12-15T22:56:09.084 DEBUG:teuthology.orchestra.run.smithi072:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-15T22:56:09.162 INFO:teuthology.orchestra.run.smithi072.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-15T22:56:09.162 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-15T22:56:09.162 INFO:teuthology.orchestra.run.smithi072.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-15T22:56:09.162 INFO:teuthology.orchestra.run.smithi072.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-15T22:56:09.162 INFO:teuthology.orchestra.run.smithi072.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-15T22:56:09.162 INFO:teuthology.orchestra.run.smithi072.stdout: = sunit=0 swidth=0 blks 2023-12-15T22:56:09.162 INFO:teuthology.orchestra.run.smithi072.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-15T22:56:09.163 INFO:teuthology.orchestra.run.smithi072.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-15T22:56:09.163 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-15T22:56:09.163 INFO:teuthology.orchestra.run.smithi072.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-15T22:56:09.671 INFO:teuthology.orchestra.run.smithi072.stdout:Discarding blocks...Done. 2023-12-15T22:56:09.673 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi072.front.sepia.ceph.com -o noatime 2023-12-15T22:56:09.673 DEBUG:teuthology.orchestra.run.smithi072:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2023-12-15T22:56:09.817 DEBUG:teuthology.orchestra.run.smithi072:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-12-15T22:56:09.853 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-12-15T22:56:09.923 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2023-12-15T22:56:09.924 INFO:tasks.ceph:role: osd.3 2023-12-15T22:56:09.924 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi072.front.sepia.ceph.com 2023-12-15T22:56:09.924 DEBUG:teuthology.orchestra.run.smithi072:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-15T22:56:10.000 INFO:teuthology.orchestra.run.smithi072.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-15T22:56:10.000 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-15T22:56:10.000 INFO:teuthology.orchestra.run.smithi072.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-15T22:56:10.000 INFO:teuthology.orchestra.run.smithi072.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-15T22:56:10.000 INFO:teuthology.orchestra.run.smithi072.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-15T22:56:10.000 INFO:teuthology.orchestra.run.smithi072.stdout: = sunit=0 swidth=0 blks 2023-12-15T22:56:10.001 INFO:teuthology.orchestra.run.smithi072.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-15T22:56:10.001 INFO:teuthology.orchestra.run.smithi072.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-15T22:56:10.001 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-15T22:56:10.001 INFO:teuthology.orchestra.run.smithi072.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-15T22:56:10.448 INFO:teuthology.orchestra.run.smithi072.stdout:Discarding blocks...Done. 2023-12-15T22:56:10.450 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi072.front.sepia.ceph.com -o noatime 2023-12-15T22:56:10.450 DEBUG:teuthology.orchestra.run.smithi072:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-3 2023-12-15T22:56:10.543 DEBUG:teuthology.orchestra.run.smithi072:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-12-15T22:56:10.574 DEBUG:teuthology.orchestra.run.smithi072:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-15T22:56:11.141 INFO:teuthology.orchestra.run.smithi072.stderr:WARNING: debug mode. Not for benchmarking or production 2023-12-15T22:56:11.142 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-12-15 22:56:11,143 seastar - Reactor backend: linux-aio 2023-12-15T22:56:11.162 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-12-15 22:56:11,163 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-12-15T22:56:11.162 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-12-15 22:56:11,163 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-12-15T22:56:11.163 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-12-15 22:56:11,163 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-12-15T22:56:11.163 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-12-15 22:56:11,163 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-12-15T22:56:11.164 INFO:teuthology.orchestra.run.smithi072.stderr:==80846==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-12-15T22:56:11.170 INFO:teuthology.orchestra.run.smithi072.stderr:==80846==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffc75143000; bottom 0x7f9ed9a75000; size: 0x005d9b6ce000 (402039562240) 2023-12-15T22:56:11.171 INFO:teuthology.orchestra.run.smithi072.stderr:False positive error reports may follow 2023-12-15T22:56:11.171 INFO:teuthology.orchestra.run.smithi072.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-12-15T22:56:11.197 INFO:teuthology.orchestra.run.smithi072.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2cbb822a 0x2cbb95d8 0x2cbbab26 0x2cbbaeaa 0x2cbbaff4 0x2cbbb44a 0x54daf 0x231b7ddb 0x231b88b5 0x231bf729 0x22fa5d82 0x22e65a6d 0x24eb01a0 0x24eb7956 0x24eba2a4 0x24ebb47d 0x22a014df 0x22a5213e 0x22a75c19 0x2cb5a545 0x2cbf8352 0x2ce1ce99 0x2ce1ece3 0x2c87339c 0x2c8752a5 0x1f07f316 0x1f0880b3 0x1eef64f2 0x3feaf 0x3ff5f 0x1e99fa44 2023-12-15T22:56:11.197 INFO:teuthology.orchestra.run.smithi072.stderr:kernel callstack: 2023-12-15T22:56:11.206 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-12-15 22:56:11,207 [shard 0] osd - bypassing the config fetch due to --no-mon-config 2023-12-15T22:56:11.206 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-12-15 22:56:11,207 [shard 0] osd - get_early_config: set --smp 3 2023-12-15T22:56:11.688 INFO:teuthology.orchestra.run.smithi072.stderr:----------------------------------------------------- 2023-12-15T22:56:11.688 INFO:teuthology.orchestra.run.smithi072.stderr:Suppressions used: 2023-12-15T22:56:11.688 INFO:teuthology.orchestra.run.smithi072.stderr: count bytes template 2023-12-15T22:56:11.688 INFO:teuthology.orchestra.run.smithi072.stderr: 1 8 MallocExtension::Register 2023-12-15T22:56:11.688 INFO:teuthology.orchestra.run.smithi072.stderr:----------------------------------------------------- 2023-12-15T22:56:11.688 INFO:teuthology.orchestra.run.smithi072.stderr: 2023-12-15T22:56:11.742 INFO:teuthology.orchestra.run.smithi072.stderr:WARNING: debug mode. Not for benchmarking or production 2023-12-15T22:56:11.743 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-12-15 22:56:11,744 seastar - Reactor backend: linux-aio 2023-12-15T22:56:11.761 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-12-15 22:56:11,762 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-12-15T22:56:11.761 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-12-15 22:56:11,762 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-12-15T22:56:11.761 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-12-15 22:56:11,762 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-12-15T22:56:11.768 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-12-15 22:56:11,769 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-12-15T22:56:11.771 INFO:teuthology.orchestra.run.smithi072.stderr:==80844==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-12-15T22:56:11.777 INFO:teuthology.orchestra.run.smithi072.stderr:==80844==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffc75143000; bottom 0x7f9ed785e000; size: 0x005d9d8e5000 (402075308032) 2023-12-15T22:56:11.777 INFO:teuthology.orchestra.run.smithi072.stderr:False positive error reports may follow 2023-12-15T22:56:11.777 INFO:teuthology.orchestra.run.smithi072.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-12-15T22:56:15.779 INFO:teuthology.orchestra.run.smithi072.stderr:----------------------------------------------------- 2023-12-15T22:56:15.780 INFO:teuthology.orchestra.run.smithi072.stderr:Suppressions used: 2023-12-15T22:56:15.780 INFO:teuthology.orchestra.run.smithi072.stderr: count bytes template 2023-12-15T22:56:15.780 INFO:teuthology.orchestra.run.smithi072.stderr: 1 8 MallocExtension::Register 2023-12-15T22:56:15.780 INFO:teuthology.orchestra.run.smithi072.stderr:----------------------------------------------------- 2023-12-15T22:56:15.780 INFO:teuthology.orchestra.run.smithi072.stderr: 2023-12-15T22:56:15.780 INFO:teuthology.orchestra.run.smithi072.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 6b8c5e6b-dc21-4b86-85c9-629eae1ef431 2023-12-15T22:56:15.816 DEBUG:teuthology.orchestra.run.smithi072:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-12-15T22:56:15.842 DEBUG:teuthology.orchestra.run.smithi072:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-15T22:56:16.354 INFO:teuthology.orchestra.run.smithi072.stderr:WARNING: debug mode. Not for benchmarking or production 2023-12-15T22:56:16.355 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-12-15 22:56:16,355 seastar - Reactor backend: linux-aio 2023-12-15T22:56:16.371 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-12-15 22:56:16,372 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-12-15T22:56:16.372 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-12-15 22:56:16,372 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-12-15T22:56:16.372 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-12-15 22:56:16,372 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-12-15T22:56:16.372 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-12-15 22:56:16,372 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-12-15T22:56:16.373 INFO:teuthology.orchestra.run.smithi072.stderr:==81029==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-12-15T22:56:16.379 INFO:teuthology.orchestra.run.smithi072.stderr:==81029==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffcf69ca000; bottom 0x7f5381b92000; size: 0x00a974e38000 (727810539520) 2023-12-15T22:56:16.379 INFO:teuthology.orchestra.run.smithi072.stderr:False positive error reports may follow 2023-12-15T22:56:16.380 INFO:teuthology.orchestra.run.smithi072.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-12-15T22:56:16.420 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-12-15 22:56:16,421 [shard 0] osd - bypassing the config fetch due to --no-mon-config 2023-12-15T22:56:16.420 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-12-15 22:56:16,421 [shard 0] osd - get_early_config: set --smp 3 2023-12-15T22:56:16.910 INFO:teuthology.orchestra.run.smithi072.stderr:----------------------------------------------------- 2023-12-15T22:56:16.910 INFO:teuthology.orchestra.run.smithi072.stderr:Suppressions used: 2023-12-15T22:56:16.910 INFO:teuthology.orchestra.run.smithi072.stderr: count bytes template 2023-12-15T22:56:16.910 INFO:teuthology.orchestra.run.smithi072.stderr: 1 8 MallocExtension::Register 2023-12-15T22:56:16.911 INFO:teuthology.orchestra.run.smithi072.stderr:----------------------------------------------------- 2023-12-15T22:56:16.911 INFO:teuthology.orchestra.run.smithi072.stderr: 2023-12-15T22:56:16.965 INFO:teuthology.orchestra.run.smithi072.stderr:WARNING: debug mode. Not for benchmarking or production 2023-12-15T22:56:16.966 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-12-15 22:56:16,967 seastar - Reactor backend: linux-aio 2023-12-15T22:56:16.984 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-12-15 22:56:16,985 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-12-15T22:56:16.984 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-12-15 22:56:16,985 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-12-15T22:56:16.984 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-12-15 22:56:16,985 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-12-15T22:56:16.991 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-12-15 22:56:16,992 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-12-15T22:56:16.994 INFO:teuthology.orchestra.run.smithi072.stderr:==81027==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-12-15T22:56:17.000 INFO:teuthology.orchestra.run.smithi072.stderr:==81027==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffcf69ca000; bottom 0x7f537f97b000; size: 0x00a97704f000 (727846285312) 2023-12-15T22:56:17.000 INFO:teuthology.orchestra.run.smithi072.stderr:False positive error reports may follow 2023-12-15T22:56:17.000 INFO:teuthology.orchestra.run.smithi072.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-12-15T22:56:20.988 INFO:teuthology.orchestra.run.smithi072.stderr:----------------------------------------------------- 2023-12-15T22:56:20.988 INFO:teuthology.orchestra.run.smithi072.stderr:Suppressions used: 2023-12-15T22:56:20.988 INFO:teuthology.orchestra.run.smithi072.stderr: count bytes template 2023-12-15T22:56:20.988 INFO:teuthology.orchestra.run.smithi072.stderr: 1 8 MallocExtension::Register 2023-12-15T22:56:20.988 INFO:teuthology.orchestra.run.smithi072.stderr:----------------------------------------------------- 2023-12-15T22:56:20.988 INFO:teuthology.orchestra.run.smithi072.stderr: 2023-12-15T22:56:20.989 INFO:teuthology.orchestra.run.smithi072.stdout:created object store /var/lib/ceph/osd/ceph-3 for osd.3 fsid 6b8c5e6b-dc21-4b86-85c9-629eae1ef431 2023-12-15T22:56:21.025 DEBUG:teuthology.orchestra.run.smithi072:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-12-15T22:56:21.050 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-15T22:56:21.051 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-12-15T22:56:21.051 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-12-15T22:56:21.082 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-12-15T22:56:21.082 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-12-15T22:56:21.152 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2023-12-15T22:56:21.152 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-12-15T22:56:21.179 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2023-12-15T22:56:21.179 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-12-15T22:56:21.246 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2023-12-15T22:56:21.246 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-12-15T22:56:21.313 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-12-15T22:56:21.313 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-12-15T22:56:21.331 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2023-12-15T22:56:21.331 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-12-15T22:56:21.367 INFO:tasks.ceph:Adding keys to all mons... 2023-12-15T22:56:21.368 DEBUG:teuthology.orchestra.run.smithi071:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-15T22:56:21.400 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.0] 2023-12-15T22:56:21.400 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQB/2XxlBQKXMhAA7/sqLnYce41gbYVAMpmbLA== 2023-12-15T22:56:21.401 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.1] 2023-12-15T22:56:21.401 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQCF2Xxlz9vTARAA6ZZTChPW9hoE8jflCsgX1g== 2023-12-15T22:56:21.401 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr.x] 2023-12-15T22:56:21.401 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQB92XxlfoViAhAAsaw/Fmghc51fDI9trKvIxA== 2023-12-15T22:56:21.401 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.2] 2023-12-15T22:56:21.401 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQCL2XxlgEdSMRAA0lMUfpiyOGartbPLp5b0VA== 2023-12-15T22:56:21.401 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.3] 2023-12-15T22:56:21.401 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQCR2XxlRdnrAhAAEFz40PdyNfXYBrPt2Xk0BQ== 2023-12-15T22:56:21.401 INFO:teuthology.orchestra.run.smithi071.stdout:[client.0] 2023-12-15T22:56:21.401 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQB92Xxl2CJ2BRAA95ifggYDyP9zmSoliOdojg== 2023-12-15T22:56:21.401 INFO:teuthology.orchestra.run.smithi071.stdout:[client.1] 2023-12-15T22:56:21.401 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQB92Xxl5WU+CRAAuNUfhMWTjIyOOYqv0CQr5g== 2023-12-15T22:56:21.402 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-15T22:56:21.509 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-15T22:56:21.569 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-15T22:56:21.627 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-15T22:56:21.685 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-15T22:56:21.743 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-15T22:56:21.804 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-15T22:56:21.865 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-15T22:56:21.865 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-12-15T22:56:21.891 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-15T22:56:21.986 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:21.986+0000 7f7a5781e980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:21.986 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:21.986+0000 7f7a5781e980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:22.002 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-12-15T22:56:22.028 DEBUG:teuthology.orchestra.run.smithi071:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-15T22:56:22.082 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-12-15T22:56:22.083 INFO:tasks.ceph.mon.a:Restarting daemon 2023-12-15T22:56:22.083 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-12-15T22:56:22.125 INFO:tasks.ceph.mon.a:Started 2023-12-15T22:56:22.125 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-12-15T22:56:22.125 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-12-15T22:56:22.125 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-12-15T22:56:22.127 INFO:tasks.ceph.mgr.x:Started 2023-12-15T22:56:22.128 INFO:tasks.ceph:Setting crush tunables to default 2023-12-15T22:56:22.128 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd crush tunables default 2023-12-15T22:56:22.204 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T22:56:22.203+0000 7f927de1b980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:22.204 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T22:56:22.203+0000 7f927de1b980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:22.204 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T22:56:22.203+0000 7f927de1b980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:22.221 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-12-15T22:56:22.221+0000 7f5e52408f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:22.234 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:22.228+0000 7efce166d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:22.272 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:22.272+0000 7efce166d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:22.273 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-12-15T22:56:22.273+0000 7f5e52408f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:22.273 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-12-15T22:56:22.273+0000 7f5e52408f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:22.342 INFO:teuthology.orchestra.run.smithi071.stderr:adjusted tunables profile to default 2023-12-15T22:56:22.353 INFO:tasks.ceph:check_enable_crimson: True 2023-12-15T22:56:22.353 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2023-12-15T22:56:22.353 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2023-12-15T22:56:22.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:22.451+0000 7f70e0b11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:22.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:22.459+0000 7f70e0b11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:22.536 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T22:56:22.536+0000 7f9275df1640 -1 WARNING: experimental feature 'crimson' is enabled 2023-12-15T22:56:22.536 INFO:tasks.ceph.mon.a.smithi071.stderr:Please be aware that this feature is experimental, untested, 2023-12-15T22:56:22.536 INFO:tasks.ceph.mon.a.smithi071.stderr:unsupported, and may result in data corruption, data loss, 2023-12-15T22:56:22.536 INFO:tasks.ceph.mon.a.smithi071.stderr:and/or irreparable damage to your cluster. Do not use 2023-12-15T22:56:22.537 INFO:tasks.ceph.mon.a.smithi071.stderr:feature with important data. 2023-12-15T22:56:22.537 INFO:tasks.ceph.mon.a.smithi071.stderr: 2023-12-15T22:56:22.744 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-12-15T22:56:22.744+0000 7f5e52408f00 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-15T22:56:22.796 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-12-15T22:56:22.796+0000 7f5e52408f00 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-15T22:56:23.341 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-12-15T22:56:23.341+0000 7f5e52408f00 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-15T22:56:23.358 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-12-15T22:56:23.358 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-12-15T22:56:23.358 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-12-15T22:56:23.388 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-12-15T22:56:23.388 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-12-15T22:56:23.393 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-12-15T22:56:23.393+0000 7f5e52408f00 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-15T22:56:23.438 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-12-15T22:56:23.438+0000 7f5e52408f00 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-15T22:56:23.456 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2023-12-15T22:56:23.457 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-12-15T22:56:23.483 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2023-12-15T22:56:23.483 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-12-15T22:56:23.494 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-12-15T22:56:23.493+0000 7f5e52408f00 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-15T22:56:23.519 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --cluster ceph osd new c5b5c4e6-5bf6-4ea1-a600-2b9e8c0b441d 0 2023-12-15T22:56:23.538 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-12-15T22:56:23.538+0000 7f5e52408f00 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-15T22:56:23.642 INFO:teuthology.orchestra.run.smithi072.stderr:2023-12-15T22:56:23.637+0000 7fb7e4e17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:23.645 INFO:teuthology.orchestra.run.smithi072.stderr:2023-12-15T22:56:23.645+0000 7fb7e4e17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:23.645 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-12-15T22:56:23.645+0000 7f5e52408f00 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-15T22:56:23.717 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-12-15T22:56:23.717+0000 7f5e52408f00 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-15T22:56:23.723 INFO:teuthology.orchestra.run.smithi072.stdout:0 2023-12-15T22:56:23.734 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --cluster ceph osd new 2f78d6d0-60d6-4135-a871-415a46a14b55 1 2023-12-15T22:56:23.766 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-12-15T22:56:23.766+0000 7f5e52408f00 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-15T22:56:23.821 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-12-15T22:56:23.821+0000 7f5e52408f00 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-15T22:56:23.839 INFO:teuthology.orchestra.run.smithi072.stderr:2023-12-15T22:56:23.834+0000 7f9db579e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:23.842 INFO:teuthology.orchestra.run.smithi072.stderr:2023-12-15T22:56:23.842+0000 7f9db579e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:23.871 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-12-15T22:56:23.871+0000 7f5e52408f00 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-15T22:56:23.920 INFO:teuthology.orchestra.run.smithi072.stdout:1 2023-12-15T22:56:23.931 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --cluster ceph osd new c413b63f-cfa9-481a-abbe-0824b6060779 2 2023-12-15T22:56:24.076 INFO:teuthology.orchestra.run.smithi072.stderr:2023-12-15T22:56:24.070+0000 7f7d8c51e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:24.078 INFO:teuthology.orchestra.run.smithi072.stderr:2023-12-15T22:56:24.078+0000 7f7d8c51e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:24.156 INFO:teuthology.orchestra.run.smithi072.stdout:2 2023-12-15T22:56:24.167 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --cluster ceph osd new 198dfcc4-27f8-41d8-9d0c-33dd50150e68 3 2023-12-15T22:56:24.267 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-12-15T22:56:24.267+0000 7f5e52408f00 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-15T22:56:24.271 INFO:teuthology.orchestra.run.smithi072.stderr:2023-12-15T22:56:24.266+0000 7f803ef5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:24.277 INFO:teuthology.orchestra.run.smithi072.stderr:2023-12-15T22:56:24.274+0000 7f803ef5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:24.352 INFO:teuthology.orchestra.run.smithi072.stdout:3 2023-12-15T22:56:24.363 DEBUG:teuthology.orchestra.run.smithi071:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2023-12-15T22:56:24.397 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-12-15T22:56:24.397+0000 7f5e52408f00 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-15T22:56:24.399 INFO:tasks.ceph.osd.0:Restarting daemon 2023-12-15T22:56:24.399 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2023-12-15T22:56:24.442 INFO:tasks.ceph.osd.0:Started 2023-12-15T22:56:24.442 DEBUG:teuthology.orchestra.run.smithi071:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2023-12-15T22:56:24.475 INFO:tasks.ceph.osd.1:Restarting daemon 2023-12-15T22:56:24.475 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2023-12-15T22:56:24.513 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-12-15T22:56:24.513+0000 7f5e52408f00 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-15T22:56:24.517 INFO:tasks.ceph.osd.1:Started 2023-12-15T22:56:24.517 DEBUG:teuthology.orchestra.run.smithi072:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2023-12-15T22:56:24.549 INFO:tasks.ceph.osd.2:Restarting daemon 2023-12-15T22:56:24.549 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2023-12-15T22:56:24.561 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-12-15T22:56:24.561+0000 7f5e52408f00 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-15T22:56:24.563 INFO:tasks.ceph.osd.2:Started 2023-12-15T22:56:24.563 DEBUG:teuthology.orchestra.run.smithi072:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.3.log 2023-12-15T22:56:24.594 INFO:tasks.ceph.osd.3:Restarting daemon 2023-12-15T22:56:24.594 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2023-12-15T22:56:24.637 INFO:tasks.ceph.osd.3:Started 2023-12-15T22:56:24.638 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-15T22:56:24.668 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-12-15T22:56:24.667+0000 7f5e52408f00 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-15T22:56:24.759 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:24.754+0000 7f317bcad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:24.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:24.763+0000 7f317bcad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:24.779 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-12-15T22:56:24.779+0000 7f5e52408f00 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-15T22:56:24.831 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:56:24.831 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":7,"fsid":"6b8c5e6b-dc21-4b86-85c9-629eae1ef431","created":"2023-12-15T22:56:22.261503+0000","modified":"2023-12-15T22:56:24.352746+0000","last_up_change":"0.000000","last_in_change":"2023-12-15T22:56:24.352746+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"c5b5c4e6-5bf6-4ea1-a600-2b9e8c0b441d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f78d6d0-60d6-4135-a871-415a46a14b55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c413b63f-cfa9-481a-abbe-0824b6060779","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"198dfcc4-27f8-41d8-9d0c-33dd50150e68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-15T22:56:24.844 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-12-15T22:56:24.844 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-15T22:56:24.943 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-12-15T22:56:24.943+0000 7f5e52408f00 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-15T22:56:24.991 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-12-15T22:56:24.991+0000 7f5e52408f00 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-15T22:56:25.144 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-12-15T22:56:25.143+0000 7f5e52408f00 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-15T22:56:25.247 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-15T22:56:25.345 INFO:teuthology.misc.health.smithi071.stderr:2023-12-15T22:56:25.342+0000 7f4981a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:25.350 INFO:teuthology.misc.health.smithi071.stderr:2023-12-15T22:56:25.350+0000 7f4981a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:25.417 INFO:teuthology.misc.health.smithi071.stdout: 2023-12-15T22:56:25.417 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":7,"fsid":"6b8c5e6b-dc21-4b86-85c9-629eae1ef431","created":"2023-12-15T22:56:22.261503+0000","modified":"2023-12-15T22:56:24.352746+0000","last_up_change":"0.000000","last_in_change":"2023-12-15T22:56:24.352746+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"c5b5c4e6-5bf6-4ea1-a600-2b9e8c0b441d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f78d6d0-60d6-4135-a871-415a46a14b55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c413b63f-cfa9-481a-abbe-0824b6060779","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"198dfcc4-27f8-41d8-9d0c-33dd50150e68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-15T22:56:25.427 DEBUG:teuthology.misc:0 of 4 OSDs are up 2023-12-15T22:56:26.051 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-12-15T22:56:26.050+0000 7f5e52408f00 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-15T22:56:26.334 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-12-15T22:56:26.334+0000 7f5e52408f00 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-15T22:56:26.387 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-12-15T22:56:26.387+0000 7f5e52408f00 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-15T22:56:26.493 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-12-15T22:56:26.493+0000 7f5e52408f00 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-15T22:56:27.517 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-12-15T22:56:27.516+0000 7f5df2630640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-12-15T22:56:31.830 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-15T22:56:31.936 INFO:teuthology.misc.health.smithi071.stderr:2023-12-15T22:56:31.933+0000 7fc2e5f15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:31.939 INFO:teuthology.misc.health.smithi071.stderr:2023-12-15T22:56:31.939+0000 7fc2e5f15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:32.118 INFO:teuthology.misc.health.smithi071.stdout: 2023-12-15T22:56:32.118 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"6b8c5e6b-dc21-4b86-85c9-629eae1ef431","created":"2023-12-15T22:56:22.261503+0000","modified":"2023-12-15T22:56:31.525572+0000","last_up_change":"2023-12-15T22:56:29.518779+0000","last_in_change":"2023-12-15T22:56:24.352746+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-15T22:56:30.553423+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c5b5c4e6-5bf6-4ea1-a600-2b9e8c0b441d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6801","nonce":1516130097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":1516130097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":1516130097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6805","nonce":1516130097}]},"public_addr":"172.21.15.71:6801/1516130097","cluster_addr":"172.21.15.71:6800/1516130097","heartbeat_back_addr":"172.21.15.71:6804/1516130097","heartbeat_front_addr":"172.21.15.71:6805/1516130097","state":["exists","up"]},{"osd":1,"uuid":"2f78d6d0-60d6-4135-a871-415a46a14b55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":913345433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6803","nonce":913345433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":913345433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6807","nonce":913345433}]},"public_addr":"172.21.15.71:6802/913345433","cluster_addr":"172.21.15.71:6803/913345433","heartbeat_back_addr":"172.21.15.71:6806/913345433","heartbeat_front_addr":"172.21.15.71:6807/913345433","state":["exists","up"]},{"osd":2,"uuid":"c413b63f-cfa9-481a-abbe-0824b6060779","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":3580816619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6805","nonce":3580816619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6809","nonce":3580816619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3580816619}]},"public_addr":"172.21.15.72:6804/3580816619","cluster_addr":"172.21.15.72:6805/3580816619","heartbeat_back_addr":"172.21.15.72:6809/3580816619","heartbeat_front_addr":"172.21.15.72:6808/3580816619","state":["exists","up"]},{"osd":3,"uuid":"198dfcc4-27f8-41d8-9d0c-33dd50150e68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6803","nonce":628223927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":628223927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6807","nonce":628223927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":628223927}]},"public_addr":"172.21.15.72:6803/628223927","cluster_addr":"172.21.15.72:6802/628223927","heartbeat_back_addr":"172.21.15.72:6807/628223927","heartbeat_front_addr":"172.21.15.72:6806/628223927","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-15T22:56:32.128 DEBUG:teuthology.misc:4 of 4 OSDs are up 2023-12-15T22:56:32.129 INFO:tasks.ceph:Creating RBD pool 2023-12-15T22:56:32.129 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd pool create rbd 8 2023-12-15T22:56:32.233 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:32.227+0000 7f230366d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:32.236 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:32.236+0000 7f230366d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:32.530 INFO:teuthology.orchestra.run.smithi071.stderr:pool 'rbd' created 2023-12-15T22:56:32.544 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-12-15T22:56:32.662 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:32.654+0000 7f31dd66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:32.665 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:32.665+0000 7f31dd66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:33.534 INFO:teuthology.orchestra.run.smithi071.stderr:enabled application 'rbd' on pool 'rbd' 2023-12-15T22:56:33.550 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-12-15T22:56:33.550 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-12-15T22:56:33.550 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-12-15T22:56:33.551 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-12-15T22:56:33.551 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-15T22:56:33.679 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:33.675+0000 7f28a179e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:33.683 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:33.683+0000 7f28a179e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:33.918 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:56:33.930 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":4,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":655724491},{"type":"v1","addr":"172.21.15.72:6801","nonce":655724491}]},"active_addr":"172.21.15.72:6801/655724491","active_change":"2023-12-15T22:56:26.507987+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.72:0","nonce":339929012}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.72:0","nonce":3090042757}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.72:0","nonce":1755677401}]}]} 2023-12-15T22:56:33.932 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-12-15T22:56:33.932 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-12-15T22:56:33.932 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-15T22:56:34.037 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:34.034+0000 7f1f3291e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:34.040 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:34.040+0000 7f1f3291e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:34.215 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:56:34.215 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":12,"fsid":"6b8c5e6b-dc21-4b86-85c9-629eae1ef431","created":"2023-12-15T22:56:22.261503+0000","modified":"2023-12-15T22:56:33.534083+0000","last_up_change":"2023-12-15T22:56:29.518779+0000","last_in_change":"2023-12-15T22:56:24.352746+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-15T22:56:30.553423+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-15T22:56:32.428134+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c5b5c4e6-5bf6-4ea1-a600-2b9e8c0b441d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6801","nonce":1516130097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":1516130097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":1516130097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6805","nonce":1516130097}]},"public_addr":"172.21.15.71:6801/1516130097","cluster_addr":"172.21.15.71:6800/1516130097","heartbeat_back_addr":"172.21.15.71:6804/1516130097","heartbeat_front_addr":"172.21.15.71:6805/1516130097","state":["exists","up"]},{"osd":1,"uuid":"2f78d6d0-60d6-4135-a871-415a46a14b55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":913345433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6803","nonce":913345433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":913345433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6807","nonce":913345433}]},"public_addr":"172.21.15.71:6802/913345433","cluster_addr":"172.21.15.71:6803/913345433","heartbeat_back_addr":"172.21.15.71:6806/913345433","heartbeat_front_addr":"172.21.15.71:6807/913345433","state":["exists","up"]},{"osd":2,"uuid":"c413b63f-cfa9-481a-abbe-0824b6060779","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":3580816619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6805","nonce":3580816619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6809","nonce":3580816619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3580816619}]},"public_addr":"172.21.15.72:6804/3580816619","cluster_addr":"172.21.15.72:6805/3580816619","heartbeat_back_addr":"172.21.15.72:6809/3580816619","heartbeat_front_addr":"172.21.15.72:6808/3580816619","state":["exists","up"]},{"osd":3,"uuid":"198dfcc4-27f8-41d8-9d0c-33dd50150e68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6803","nonce":628223927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":628223927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6807","nonce":628223927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":628223927}]},"public_addr":"172.21.15.72:6803/628223927","cluster_addr":"172.21.15.72:6802/628223927","heartbeat_back_addr":"172.21.15.72:6807/628223927","heartbeat_front_addr":"172.21.15.72:6806/628223927","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-15T22:56:34.227 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-12-15T22:56:34.227 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-15T22:56:34.332 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:34.328+0000 7efdad15e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:34.335 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:34.335+0000 7efdad15e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:34.510 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:56:34.510 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":12,"fsid":"6b8c5e6b-dc21-4b86-85c9-629eae1ef431","created":"2023-12-15T22:56:22.261503+0000","modified":"2023-12-15T22:56:33.534083+0000","last_up_change":"2023-12-15T22:56:29.518779+0000","last_in_change":"2023-12-15T22:56:24.352746+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-15T22:56:30.553423+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-15T22:56:32.428134+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c5b5c4e6-5bf6-4ea1-a600-2b9e8c0b441d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6801","nonce":1516130097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":1516130097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":1516130097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6805","nonce":1516130097}]},"public_addr":"172.21.15.71:6801/1516130097","cluster_addr":"172.21.15.71:6800/1516130097","heartbeat_back_addr":"172.21.15.71:6804/1516130097","heartbeat_front_addr":"172.21.15.71:6805/1516130097","state":["exists","up"]},{"osd":1,"uuid":"2f78d6d0-60d6-4135-a871-415a46a14b55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":913345433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6803","nonce":913345433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":913345433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6807","nonce":913345433}]},"public_addr":"172.21.15.71:6802/913345433","cluster_addr":"172.21.15.71:6803/913345433","heartbeat_back_addr":"172.21.15.71:6806/913345433","heartbeat_front_addr":"172.21.15.71:6807/913345433","state":["exists","up"]},{"osd":2,"uuid":"c413b63f-cfa9-481a-abbe-0824b6060779","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":3580816619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6805","nonce":3580816619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6809","nonce":3580816619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3580816619}]},"public_addr":"172.21.15.72:6804/3580816619","cluster_addr":"172.21.15.72:6805/3580816619","heartbeat_back_addr":"172.21.15.72:6809/3580816619","heartbeat_front_addr":"172.21.15.72:6808/3580816619","state":["exists","up"]},{"osd":3,"uuid":"198dfcc4-27f8-41d8-9d0c-33dd50150e68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6803","nonce":628223927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":628223927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6807","nonce":628223927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":628223927}]},"public_addr":"172.21.15.72:6803/628223927","cluster_addr":"172.21.15.72:6802/628223927","heartbeat_back_addr":"172.21.15.72:6807/628223927","heartbeat_front_addr":"172.21.15.72:6806/628223927","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-15T22:56:34.522 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-12-15T22:56:34.523 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-12-15T22:56:34.523 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-12-15T22:56:34.524 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-12-15T22:56:34.645 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:34.641+0000 7f9ba5f5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:34.646 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:34.641+0000 7fe948d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:34.649 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:34.649+0000 7fe948d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:34.650 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:34.649+0000 7f9ba5f5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:34.661 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:34.656+0000 7fdcffa06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:34.664 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:34.664+0000 7fdcffa06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:34.666 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:34.658+0000 7f12e019e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:34.671 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:34.671+0000 7f12e019e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:34.679 INFO:teuthology.orchestra.run.smithi071.stdout:38654705669 2023-12-15T22:56:34.680 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-15T22:56:34.682 INFO:teuthology.orchestra.run.smithi071.stdout:38654705669 2023-12-15T22:56:34.683 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-15T22:56:34.692 INFO:teuthology.orchestra.run.smithi071.stdout:38654705669 2023-12-15T22:56:34.693 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-15T22:56:34.696 INFO:teuthology.orchestra.run.smithi071.stdout:38654705669 2023-12-15T22:56:34.696 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-15T22:56:34.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:34.786+0000 7fb73ad1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:34.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:34.787+0000 7fa87499e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:34.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:34.795+0000 7fb73ad1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:34.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:34.795+0000 7fa87499e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:34.849 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:34.845+0000 7ff8b0b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:34.852 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:34.852+0000 7ff8b0b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:34.867 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:34.863+0000 7f39d08ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:34.871 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:34.871+0000 7f39d08ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:34.990 INFO:teuthology.orchestra.run.smithi071.stdout:38654705669 2023-12-15T22:56:35.002 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705669 for osd.1 2023-12-15T22:56:35.002 DEBUG:teuthology.parallel:result is None 2023-12-15T22:56:35.033 INFO:teuthology.orchestra.run.smithi071.stdout:38654705669 2023-12-15T22:56:35.041 INFO:teuthology.orchestra.run.smithi071.stdout:38654705669 2023-12-15T22:56:35.047 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705669 for osd.0 2023-12-15T22:56:35.047 DEBUG:teuthology.parallel:result is None 2023-12-15T22:56:35.054 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705669 for osd.2 2023-12-15T22:56:35.054 DEBUG:teuthology.parallel:result is None 2023-12-15T22:56:35.081 INFO:teuthology.orchestra.run.smithi071.stdout:38654705669 2023-12-15T22:56:35.093 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705669 for osd.3 2023-12-15T22:56:35.093 DEBUG:teuthology.parallel:result is None 2023-12-15T22:56:35.093 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-15T22:56:35.094 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T22:56:35.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:35.234+0000 7f30ecb5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:35.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:35.240+0000 7f30ecb5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:35.416 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:56:35.416 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-12-15T22:56:35.428 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2023-12-15T22:56:34.681251+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107720,"kb_used_data":388,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379640,"statfs":{"total":386547056640,"available":386436751360,"internally_reserved":0,"allocated":397312,"data_stored":1840276,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-15T22:56:34.562104+0000","last_change":"2023-12-15T22:56:33.627005+0000","last_active":"2023-12-15T22:56:34.562104+0000","last_peered":"2023-12-15T22:56:34.562104+0000","last_clean":"2023-12-15T22:56:34.562104+0000","last_became_active":"2023-12-15T22:56:33.622181+0000","last_became_peered":"2023-12-15T22:56:33.622181+0000","last_unstale":"2023-12-15T22:56:34.562104+0000","last_undegraded":"2023-12-15T22:56:34.562104+0000","last_fullsized":"2023-12-15T22:56:34.562104+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-15T22:56:33.631817+0000","last_change":"2023-12-15T22:56:33.632348+0000","last_active":"2023-12-15T22:56:33.631817+0000","last_peered":"2023-12-15T22:56:33.631817+0000","last_clean":"2023-12-15T22:56:32.529733+0000","last_became_active":"2023-12-15T22:56:33.629022+0000","last_became_peered":"2023-12-15T22:56:33.629022+0000","last_unstale":"2023-12-15T22:56:33.631817+0000","last_undegraded":"2023-12-15T22:56:33.631817+0000","last_fullsized":"2023-12-15T22:56:33.631817+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-15T22:56:33.655368+0000","last_change":"2023-12-15T22:56:33.655736+0000","last_active":"2023-12-15T22:56:33.655368+0000","last_peered":"2023-12-15T22:56:33.655368+0000","last_clean":"2023-12-15T22:56:32.529733+0000","last_became_active":"2023-12-15T22:56:33.652917+0000","last_became_peered":"2023-12-15T22:56:33.652917+0000","last_unstale":"2023-12-15T22:56:33.655368+0000","last_undegraded":"2023-12-15T22:56:33.655368+0000","last_fullsized":"2023-12-15T22:56:33.655368+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-15T22:56:33.633514+0000","last_change":"2023-12-15T22:56:33.633890+0000","last_active":"2023-12-15T22:56:33.633514+0000","last_peered":"2023-12-15T22:56:33.633514+0000","last_clean":"2023-12-15T22:56:32.529733+0000","last_became_active":"2023-12-15T22:56:33.631134+0000","last_became_peered":"2023-12-15T22:56:33.631134+0000","last_unstale":"2023-12-15T22:56:33.633514+0000","last_undegraded":"2023-12-15T22:56:33.633514+0000","last_fullsized":"2023-12-15T22:56:33.633514+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-15T22:56:34.564536+0000","last_change":"2023-12-15T22:56:33.648541+0000","last_active":"2023-12-15T22:56:34.564536+0000","last_peered":"2023-12-15T22:56:34.564536+0000","last_clean":"2023-12-15T22:56:34.564536+0000","last_became_active":"2023-12-15T22:56:33.643681+0000","last_became_peered":"2023-12-15T22:56:33.643681+0000","last_unstale":"2023-12-15T22:56:34.564536+0000","last_undegraded":"2023-12-15T22:56:34.564536+0000","last_fullsized":"2023-12-15T22:56:34.564536+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-15T22:56:34.562623+0000","last_change":"2023-12-15T22:56:33.600485+0000","last_active":"2023-12-15T22:56:34.562623+0000","last_peered":"2023-12-15T22:56:34.562623+0000","last_clean":"2023-12-15T22:56:34.562623+0000","last_became_active":"2023-12-15T22:56:33.597320+0000","last_became_peered":"2023-12-15T22:56:33.597320+0000","last_unstale":"2023-12-15T22:56:34.562623+0000","last_undegraded":"2023-12-15T22:56:34.562623+0000","last_fullsized":"2023-12-15T22:56:34.562623+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-15T22:56:34.568914+0000","last_change":"2023-12-15T22:56:33.655797+0000","last_active":"2023-12-15T22:56:34.568914+0000","last_peered":"2023-12-15T22:56:34.568914+0000","last_clean":"2023-12-15T22:56:34.568914+0000","last_became_active":"2023-12-15T22:56:33.652885+0000","last_became_peered":"2023-12-15T22:56:33.652885+0000","last_unstale":"2023-12-15T22:56:34.568914+0000","last_undegraded":"2023-12-15T22:56:34.568914+0000","last_fullsized":"2023-12-15T22:56:34.568914+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-15T22:56:34.564319+0000","last_change":"2023-12-15T22:56:33.627481+0000","last_active":"2023-12-15T22:56:34.564319+0000","last_peered":"2023-12-15T22:56:34.564319+0000","last_clean":"2023-12-15T22:56:34.564319+0000","last_became_active":"2023-12-15T22:56:33.624689+0000","last_became_peered":"2023-12-15T22:56:33.624689+0000","last_unstale":"2023-12-15T22:56:34.564319+0000","last_undegraded":"2023-12-15T22:56:34.564319+0000","last_fullsized":"2023-12-15T22:56:34.564319+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'96","reported_seq":138,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-15T22:56:33.958795+0000","last_change":"2023-12-15T22:56:32.607017+0000","last_active":"2023-12-15T22:56:33.958795+0000","last_peered":"2023-12-15T22:56:33.958795+0000","last_clean":"2023-12-15T22:56:33.958795+0000","last_became_active":"2023-12-15T22:56:32.602384+0000","last_became_peered":"2023-12-15T22:56:32.602384+0000","last_unstale":"2023-12-15T22:56:33.958795+0000","last_undegraded":"2023-12-15T22:56:33.958795+0000","last_fullsized":"2023-12-15T22:56:33.958795+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_clean_scrub_stamp":"2023-12-15T22:56:31.525572+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26948,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344892,"statfs":{"total":96636764160,"available":96609169408,"internally_reserved":0,"allocated":114688,"data_stored":607661,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26948,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344892,"statfs":{"total":96636764160,"available":96609169408,"internally_reserved":0,"allocated":114688,"data_stored":607661,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26876,"kb_used_data":52,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344964,"statfs":{"total":96636764160,"available":96609243136,"internally_reserved":0,"allocated":53248,"data_stored":17293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26948,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344892,"statfs":{"total":96636764160,"available":96609169408,"internally_reserved":0,"allocated":114688,"data_stored":607661,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-15T22:56:35.429 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T22:56:35.533 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:35.530+0000 7f337cead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:35.536 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:35.536+0000 7f337cead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:35.710 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:56:35.710 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-12-15T22:56:35.722 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2023-12-15T22:56:34.681251+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107720,"kb_used_data":388,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379640,"statfs":{"total":386547056640,"available":386436751360,"internally_reserved":0,"allocated":397312,"data_stored":1840276,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-15T22:56:34.562104+0000","last_change":"2023-12-15T22:56:33.627005+0000","last_active":"2023-12-15T22:56:34.562104+0000","last_peered":"2023-12-15T22:56:34.562104+0000","last_clean":"2023-12-15T22:56:34.562104+0000","last_became_active":"2023-12-15T22:56:33.622181+0000","last_became_peered":"2023-12-15T22:56:33.622181+0000","last_unstale":"2023-12-15T22:56:34.562104+0000","last_undegraded":"2023-12-15T22:56:34.562104+0000","last_fullsized":"2023-12-15T22:56:34.562104+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-15T22:56:33.631817+0000","last_change":"2023-12-15T22:56:33.632348+0000","last_active":"2023-12-15T22:56:33.631817+0000","last_peered":"2023-12-15T22:56:33.631817+0000","last_clean":"2023-12-15T22:56:32.529733+0000","last_became_active":"2023-12-15T22:56:33.629022+0000","last_became_peered":"2023-12-15T22:56:33.629022+0000","last_unstale":"2023-12-15T22:56:33.631817+0000","last_undegraded":"2023-12-15T22:56:33.631817+0000","last_fullsized":"2023-12-15T22:56:33.631817+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-15T22:56:33.655368+0000","last_change":"2023-12-15T22:56:33.655736+0000","last_active":"2023-12-15T22:56:33.655368+0000","last_peered":"2023-12-15T22:56:33.655368+0000","last_clean":"2023-12-15T22:56:32.529733+0000","last_became_active":"2023-12-15T22:56:33.652917+0000","last_became_peered":"2023-12-15T22:56:33.652917+0000","last_unstale":"2023-12-15T22:56:33.655368+0000","last_undegraded":"2023-12-15T22:56:33.655368+0000","last_fullsized":"2023-12-15T22:56:33.655368+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-15T22:56:33.633514+0000","last_change":"2023-12-15T22:56:33.633890+0000","last_active":"2023-12-15T22:56:33.633514+0000","last_peered":"2023-12-15T22:56:33.633514+0000","last_clean":"2023-12-15T22:56:32.529733+0000","last_became_active":"2023-12-15T22:56:33.631134+0000","last_became_peered":"2023-12-15T22:56:33.631134+0000","last_unstale":"2023-12-15T22:56:33.633514+0000","last_undegraded":"2023-12-15T22:56:33.633514+0000","last_fullsized":"2023-12-15T22:56:33.633514+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-15T22:56:34.564536+0000","last_change":"2023-12-15T22:56:33.648541+0000","last_active":"2023-12-15T22:56:34.564536+0000","last_peered":"2023-12-15T22:56:34.564536+0000","last_clean":"2023-12-15T22:56:34.564536+0000","last_became_active":"2023-12-15T22:56:33.643681+0000","last_became_peered":"2023-12-15T22:56:33.643681+0000","last_unstale":"2023-12-15T22:56:34.564536+0000","last_undegraded":"2023-12-15T22:56:34.564536+0000","last_fullsized":"2023-12-15T22:56:34.564536+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-15T22:56:34.562623+0000","last_change":"2023-12-15T22:56:33.600485+0000","last_active":"2023-12-15T22:56:34.562623+0000","last_peered":"2023-12-15T22:56:34.562623+0000","last_clean":"2023-12-15T22:56:34.562623+0000","last_became_active":"2023-12-15T22:56:33.597320+0000","last_became_peered":"2023-12-15T22:56:33.597320+0000","last_unstale":"2023-12-15T22:56:34.562623+0000","last_undegraded":"2023-12-15T22:56:34.562623+0000","last_fullsized":"2023-12-15T22:56:34.562623+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-15T22:56:34.568914+0000","last_change":"2023-12-15T22:56:33.655797+0000","last_active":"2023-12-15T22:56:34.568914+0000","last_peered":"2023-12-15T22:56:34.568914+0000","last_clean":"2023-12-15T22:56:34.568914+0000","last_became_active":"2023-12-15T22:56:33.652885+0000","last_became_peered":"2023-12-15T22:56:33.652885+0000","last_unstale":"2023-12-15T22:56:34.568914+0000","last_undegraded":"2023-12-15T22:56:34.568914+0000","last_fullsized":"2023-12-15T22:56:34.568914+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-15T22:56:34.564319+0000","last_change":"2023-12-15T22:56:33.627481+0000","last_active":"2023-12-15T22:56:34.564319+0000","last_peered":"2023-12-15T22:56:34.564319+0000","last_clean":"2023-12-15T22:56:34.564319+0000","last_became_active":"2023-12-15T22:56:33.624689+0000","last_became_peered":"2023-12-15T22:56:33.624689+0000","last_unstale":"2023-12-15T22:56:34.564319+0000","last_undegraded":"2023-12-15T22:56:34.564319+0000","last_fullsized":"2023-12-15T22:56:34.564319+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'96","reported_seq":138,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-15T22:56:33.958795+0000","last_change":"2023-12-15T22:56:32.607017+0000","last_active":"2023-12-15T22:56:33.958795+0000","last_peered":"2023-12-15T22:56:33.958795+0000","last_clean":"2023-12-15T22:56:33.958795+0000","last_became_active":"2023-12-15T22:56:32.602384+0000","last_became_peered":"2023-12-15T22:56:32.602384+0000","last_unstale":"2023-12-15T22:56:33.958795+0000","last_undegraded":"2023-12-15T22:56:33.958795+0000","last_fullsized":"2023-12-15T22:56:33.958795+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_clean_scrub_stamp":"2023-12-15T22:56:31.525572+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26948,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344892,"statfs":{"total":96636764160,"available":96609169408,"internally_reserved":0,"allocated":114688,"data_stored":607661,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26948,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344892,"statfs":{"total":96636764160,"available":96609169408,"internally_reserved":0,"allocated":114688,"data_stored":607661,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26876,"kb_used_data":52,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344964,"statfs":{"total":96636764160,"available":96609243136,"internally_reserved":0,"allocated":53248,"data_stored":17293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26948,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344892,"statfs":{"total":96636764160,"available":96609169408,"internally_reserved":0,"allocated":114688,"data_stored":607661,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-15T22:56:35.722 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-15T22:56:35.722 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-12-15T22:56:35.723 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-12-15T22:56:35.723 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-12-15T22:56:35.826 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:35.824+0000 7fa68ec6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:35.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:35.829+0000 7fa68ec6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:36.039 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:56:36.039 INFO:teuthology.orchestra.run.smithi071.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-15T22:56:36.052 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-12-15T22:56:36.052 INFO:teuthology.run_tasks:Running task thrashosds... 2023-12-15T22:56:36.063 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_down': 0, 'chance_inject_pause_short': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2023-12-15T22:56:36.063 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-sjust-crimson-testing-2023-12-15'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'sha1': 'cd750a7fc894b5dea3d055d595c0da884f63f94f', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': 'cd750a7fc894b5dea3d055d595c0da884f63f94f'}}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'wip-sjust-crimson-testing-2023-12-15', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'cd750a7fc894b5dea3d055d595c0da884f63f94f'}} 2023-12-15T22:56:36.063 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_down': 0, 'chance_inject_pause_short': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 4, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2023-12-15T22:56:36.063 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2023-12-15T22:56:36.171 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:36.165+0000 7f7d0cd15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:36.174 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:36.174+0000 7f7d0cd15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:36.367 INFO:tasks.thrashosds:Beginning thrashosds... 2023-12-15T22:56:36.367 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-15T22:56:36.367 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T22:56:36.471 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:36.468+0000 7fc8e611e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:36.474 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:36.474+0000 7fc8e611e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:36.647 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:56:36.648 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-12-15T22:56:36.659 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-12-15T22:56:36.512307+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107720,"kb_used_data":388,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379640,"statfs":{"total":386547056640,"available":386436751360,"internally_reserved":0,"allocated":397312,"data_stored":1840276,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.831056"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-15T22:56:34.562104+0000","last_change":"2023-12-15T22:56:33.627005+0000","last_active":"2023-12-15T22:56:34.562104+0000","last_peered":"2023-12-15T22:56:34.562104+0000","last_clean":"2023-12-15T22:56:34.562104+0000","last_became_active":"2023-12-15T22:56:33.622181+0000","last_became_peered":"2023-12-15T22:56:33.622181+0000","last_unstale":"2023-12-15T22:56:34.562104+0000","last_undegraded":"2023-12-15T22:56:34.562104+0000","last_fullsized":"2023-12-15T22:56:34.562104+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-15T22:56:33.631817+0000","last_change":"2023-12-15T22:56:33.632348+0000","last_active":"2023-12-15T22:56:33.631817+0000","last_peered":"2023-12-15T22:56:33.631817+0000","last_clean":"2023-12-15T22:56:32.529733+0000","last_became_active":"2023-12-15T22:56:33.629022+0000","last_became_peered":"2023-12-15T22:56:33.629022+0000","last_unstale":"2023-12-15T22:56:33.631817+0000","last_undegraded":"2023-12-15T22:56:33.631817+0000","last_fullsized":"2023-12-15T22:56:33.631817+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-15T22:56:33.655368+0000","last_change":"2023-12-15T22:56:33.655736+0000","last_active":"2023-12-15T22:56:33.655368+0000","last_peered":"2023-12-15T22:56:33.655368+0000","last_clean":"2023-12-15T22:56:32.529733+0000","last_became_active":"2023-12-15T22:56:33.652917+0000","last_became_peered":"2023-12-15T22:56:33.652917+0000","last_unstale":"2023-12-15T22:56:33.655368+0000","last_undegraded":"2023-12-15T22:56:33.655368+0000","last_fullsized":"2023-12-15T22:56:33.655368+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-15T22:56:33.633514+0000","last_change":"2023-12-15T22:56:33.633890+0000","last_active":"2023-12-15T22:56:33.633514+0000","last_peered":"2023-12-15T22:56:33.633514+0000","last_clean":"2023-12-15T22:56:32.529733+0000","last_became_active":"2023-12-15T22:56:33.631134+0000","last_became_peered":"2023-12-15T22:56:33.631134+0000","last_unstale":"2023-12-15T22:56:33.633514+0000","last_undegraded":"2023-12-15T22:56:33.633514+0000","last_fullsized":"2023-12-15T22:56:33.633514+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-15T22:56:34.564536+0000","last_change":"2023-12-15T22:56:33.648541+0000","last_active":"2023-12-15T22:56:34.564536+0000","last_peered":"2023-12-15T22:56:34.564536+0000","last_clean":"2023-12-15T22:56:34.564536+0000","last_became_active":"2023-12-15T22:56:33.643681+0000","last_became_peered":"2023-12-15T22:56:33.643681+0000","last_unstale":"2023-12-15T22:56:34.564536+0000","last_undegraded":"2023-12-15T22:56:34.564536+0000","last_fullsized":"2023-12-15T22:56:34.564536+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-15T22:56:34.562623+0000","last_change":"2023-12-15T22:56:33.600485+0000","last_active":"2023-12-15T22:56:34.562623+0000","last_peered":"2023-12-15T22:56:34.562623+0000","last_clean":"2023-12-15T22:56:34.562623+0000","last_became_active":"2023-12-15T22:56:33.597320+0000","last_became_peered":"2023-12-15T22:56:33.597320+0000","last_unstale":"2023-12-15T22:56:34.562623+0000","last_undegraded":"2023-12-15T22:56:34.562623+0000","last_fullsized":"2023-12-15T22:56:34.562623+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-15T22:56:34.568914+0000","last_change":"2023-12-15T22:56:33.655797+0000","last_active":"2023-12-15T22:56:34.568914+0000","last_peered":"2023-12-15T22:56:34.568914+0000","last_clean":"2023-12-15T22:56:34.568914+0000","last_became_active":"2023-12-15T22:56:33.652885+0000","last_became_peered":"2023-12-15T22:56:33.652885+0000","last_unstale":"2023-12-15T22:56:34.568914+0000","last_undegraded":"2023-12-15T22:56:34.568914+0000","last_fullsized":"2023-12-15T22:56:34.568914+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-15T22:56:34.564319+0000","last_change":"2023-12-15T22:56:33.627481+0000","last_active":"2023-12-15T22:56:34.564319+0000","last_peered":"2023-12-15T22:56:34.564319+0000","last_clean":"2023-12-15T22:56:34.564319+0000","last_became_active":"2023-12-15T22:56:33.624689+0000","last_became_peered":"2023-12-15T22:56:33.624689+0000","last_unstale":"2023-12-15T22:56:34.564319+0000","last_undegraded":"2023-12-15T22:56:34.564319+0000","last_fullsized":"2023-12-15T22:56:34.564319+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'96","reported_seq":138,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-15T22:56:33.958795+0000","last_change":"2023-12-15T22:56:32.607017+0000","last_active":"2023-12-15T22:56:33.958795+0000","last_peered":"2023-12-15T22:56:33.958795+0000","last_clean":"2023-12-15T22:56:33.958795+0000","last_became_active":"2023-12-15T22:56:32.602384+0000","last_became_peered":"2023-12-15T22:56:32.602384+0000","last_unstale":"2023-12-15T22:56:33.958795+0000","last_undegraded":"2023-12-15T22:56:33.958795+0000","last_fullsized":"2023-12-15T22:56:33.958795+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_clean_scrub_stamp":"2023-12-15T22:56:31.525572+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26948,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344892,"statfs":{"total":96636764160,"available":96609169408,"internally_reserved":0,"allocated":114688,"data_stored":607661,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26948,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344892,"statfs":{"total":96636764160,"available":96609169408,"internally_reserved":0,"allocated":114688,"data_stored":607661,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26876,"kb_used_data":52,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344964,"statfs":{"total":96636764160,"available":96609243136,"internally_reserved":0,"allocated":53248,"data_stored":17293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26948,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344892,"statfs":{"total":96636764160,"available":96609169408,"internally_reserved":0,"allocated":114688,"data_stored":607661,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-15T22:56:36.660 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T22:56:36.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:36.759+0000 7f5c7266d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:36.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:36.766+0000 7f5c7266d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:36.940 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:56:36.941 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-12-15T22:56:36.952 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-12-15T22:56:36.512307+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107720,"kb_used_data":388,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379640,"statfs":{"total":386547056640,"available":386436751360,"internally_reserved":0,"allocated":397312,"data_stored":1840276,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.831056"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-15T22:56:34.562104+0000","last_change":"2023-12-15T22:56:33.627005+0000","last_active":"2023-12-15T22:56:34.562104+0000","last_peered":"2023-12-15T22:56:34.562104+0000","last_clean":"2023-12-15T22:56:34.562104+0000","last_became_active":"2023-12-15T22:56:33.622181+0000","last_became_peered":"2023-12-15T22:56:33.622181+0000","last_unstale":"2023-12-15T22:56:34.562104+0000","last_undegraded":"2023-12-15T22:56:34.562104+0000","last_fullsized":"2023-12-15T22:56:34.562104+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-15T22:56:33.631817+0000","last_change":"2023-12-15T22:56:33.632348+0000","last_active":"2023-12-15T22:56:33.631817+0000","last_peered":"2023-12-15T22:56:33.631817+0000","last_clean":"2023-12-15T22:56:32.529733+0000","last_became_active":"2023-12-15T22:56:33.629022+0000","last_became_peered":"2023-12-15T22:56:33.629022+0000","last_unstale":"2023-12-15T22:56:33.631817+0000","last_undegraded":"2023-12-15T22:56:33.631817+0000","last_fullsized":"2023-12-15T22:56:33.631817+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-15T22:56:33.655368+0000","last_change":"2023-12-15T22:56:33.655736+0000","last_active":"2023-12-15T22:56:33.655368+0000","last_peered":"2023-12-15T22:56:33.655368+0000","last_clean":"2023-12-15T22:56:32.529733+0000","last_became_active":"2023-12-15T22:56:33.652917+0000","last_became_peered":"2023-12-15T22:56:33.652917+0000","last_unstale":"2023-12-15T22:56:33.655368+0000","last_undegraded":"2023-12-15T22:56:33.655368+0000","last_fullsized":"2023-12-15T22:56:33.655368+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-15T22:56:33.633514+0000","last_change":"2023-12-15T22:56:33.633890+0000","last_active":"2023-12-15T22:56:33.633514+0000","last_peered":"2023-12-15T22:56:33.633514+0000","last_clean":"2023-12-15T22:56:32.529733+0000","last_became_active":"2023-12-15T22:56:33.631134+0000","last_became_peered":"2023-12-15T22:56:33.631134+0000","last_unstale":"2023-12-15T22:56:33.633514+0000","last_undegraded":"2023-12-15T22:56:33.633514+0000","last_fullsized":"2023-12-15T22:56:33.633514+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-15T22:56:34.564536+0000","last_change":"2023-12-15T22:56:33.648541+0000","last_active":"2023-12-15T22:56:34.564536+0000","last_peered":"2023-12-15T22:56:34.564536+0000","last_clean":"2023-12-15T22:56:34.564536+0000","last_became_active":"2023-12-15T22:56:33.643681+0000","last_became_peered":"2023-12-15T22:56:33.643681+0000","last_unstale":"2023-12-15T22:56:34.564536+0000","last_undegraded":"2023-12-15T22:56:34.564536+0000","last_fullsized":"2023-12-15T22:56:34.564536+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-15T22:56:34.562623+0000","last_change":"2023-12-15T22:56:33.600485+0000","last_active":"2023-12-15T22:56:34.562623+0000","last_peered":"2023-12-15T22:56:34.562623+0000","last_clean":"2023-12-15T22:56:34.562623+0000","last_became_active":"2023-12-15T22:56:33.597320+0000","last_became_peered":"2023-12-15T22:56:33.597320+0000","last_unstale":"2023-12-15T22:56:34.562623+0000","last_undegraded":"2023-12-15T22:56:34.562623+0000","last_fullsized":"2023-12-15T22:56:34.562623+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-15T22:56:34.568914+0000","last_change":"2023-12-15T22:56:33.655797+0000","last_active":"2023-12-15T22:56:34.568914+0000","last_peered":"2023-12-15T22:56:34.568914+0000","last_clean":"2023-12-15T22:56:34.568914+0000","last_became_active":"2023-12-15T22:56:33.652885+0000","last_became_peered":"2023-12-15T22:56:33.652885+0000","last_unstale":"2023-12-15T22:56:34.568914+0000","last_undegraded":"2023-12-15T22:56:34.568914+0000","last_fullsized":"2023-12-15T22:56:34.568914+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-15T22:56:34.564319+0000","last_change":"2023-12-15T22:56:33.627481+0000","last_active":"2023-12-15T22:56:34.564319+0000","last_peered":"2023-12-15T22:56:34.564319+0000","last_clean":"2023-12-15T22:56:34.564319+0000","last_became_active":"2023-12-15T22:56:33.624689+0000","last_became_peered":"2023-12-15T22:56:33.624689+0000","last_unstale":"2023-12-15T22:56:34.564319+0000","last_undegraded":"2023-12-15T22:56:34.564319+0000","last_fullsized":"2023-12-15T22:56:34.564319+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'96","reported_seq":138,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-15T22:56:33.958795+0000","last_change":"2023-12-15T22:56:32.607017+0000","last_active":"2023-12-15T22:56:33.958795+0000","last_peered":"2023-12-15T22:56:33.958795+0000","last_clean":"2023-12-15T22:56:33.958795+0000","last_became_active":"2023-12-15T22:56:32.602384+0000","last_became_peered":"2023-12-15T22:56:32.602384+0000","last_unstale":"2023-12-15T22:56:33.958795+0000","last_undegraded":"2023-12-15T22:56:33.958795+0000","last_fullsized":"2023-12-15T22:56:33.958795+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_clean_scrub_stamp":"2023-12-15T22:56:31.525572+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26948,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344892,"statfs":{"total":96636764160,"available":96609169408,"internally_reserved":0,"allocated":114688,"data_stored":607661,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26948,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344892,"statfs":{"total":96636764160,"available":96609169408,"internally_reserved":0,"allocated":114688,"data_stored":607661,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26876,"kb_used_data":52,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344964,"statfs":{"total":96636764160,"available":96609243136,"internally_reserved":0,"allocated":53248,"data_stored":17293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26948,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344892,"statfs":{"total":96636764160,"available":96609169408,"internally_reserved":0,"allocated":114688,"data_stored":607661,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-15T22:56:36.954 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-15T22:56:36.954 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2023-12-15T22:56:37.059 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:37.055+0000 7fb1ee35e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:37.062 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:37.061+0000 7fb1ee35e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:37.239 INFO:teuthology.orchestra.run.smithi071.stdout:epoch 13 2023-12-15T22:56:37.239 INFO:teuthology.orchestra.run.smithi071.stdout:fsid 6b8c5e6b-dc21-4b86-85c9-629eae1ef431 2023-12-15T22:56:37.239 INFO:teuthology.orchestra.run.smithi071.stdout:created 2023-12-15T22:56:22.261503+0000 2023-12-15T22:56:37.239 INFO:teuthology.orchestra.run.smithi071.stdout:modified 2023-12-15T22:56:34.539399+0000 2023-12-15T22:56:37.239 INFO:teuthology.orchestra.run.smithi071.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2023-12-15T22:56:37.239 INFO:teuthology.orchestra.run.smithi071.stdout:crush_version 3 2023-12-15T22:56:37.239 INFO:teuthology.orchestra.run.smithi071.stdout:full_ratio 0.95 2023-12-15T22:56:37.239 INFO:teuthology.orchestra.run.smithi071.stdout:backfillfull_ratio 0.9 2023-12-15T22:56:37.239 INFO:teuthology.orchestra.run.smithi071.stdout:nearfull_ratio 0.85 2023-12-15T22:56:37.239 INFO:teuthology.orchestra.run.smithi071.stdout:require_min_compat_client luminous 2023-12-15T22:56:37.239 INFO:teuthology.orchestra.run.smithi071.stdout:min_compat_client jewel 2023-12-15T22:56:37.240 INFO:teuthology.orchestra.run.smithi071.stdout:require_osd_release reef 2023-12-15T22:56:37.240 INFO:teuthology.orchestra.run.smithi071.stdout:stretch_mode_enabled false 2023-12-15T22:56:37.240 INFO:teuthology.orchestra.run.smithi071.stdout:allow_crimson=true 2023-12-15T22:56:37.240 INFO:teuthology.orchestra.run.smithi071.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 12 flags hashpspool,nopgchange,crimson stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 4.00 2023-12-15T22:56:37.240 INFO:teuthology.orchestra.run.smithi071.stdout:pool 2 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 13 flags hashpspool,nopgchange,crimson stripe_width 0 application rbd read_balance_score 1.50 2023-12-15T22:56:37.240 INFO:teuthology.orchestra.run.smithi071.stdout:max_osd 4 2023-12-15T22:56:37.240 INFO:teuthology.orchestra.run.smithi071.stdout:osd.0 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.71:6801/1516130097 v2:172.21.15.71:6800/1516130097 exists,up c5b5c4e6-5bf6-4ea1-a600-2b9e8c0b441d 2023-12-15T22:56:37.240 INFO:teuthology.orchestra.run.smithi071.stdout:osd.1 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.71:6802/913345433 v2:172.21.15.71:6803/913345433 exists,up 2f78d6d0-60d6-4135-a871-415a46a14b55 2023-12-15T22:56:37.240 INFO:teuthology.orchestra.run.smithi071.stdout:osd.2 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.72:6804/3580816619 v2:172.21.15.72:6805/3580816619 exists,up c413b63f-cfa9-481a-abbe-0824b6060779 2023-12-15T22:56:37.240 INFO:teuthology.orchestra.run.smithi071.stdout:osd.3 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.72:6803/628223927 v2:172.21.15.72:6802/628223927 exists,up 198dfcc4-27f8-41d8-9d0c-33dd50150e68 2023-12-15T22:56:37.252 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.71:6801/1516130097 v2:172.21.15.71:6800/1516130097 exists,up c5b5c4e6-5bf6-4ea1-a600-2b9e8c0b441d', 'osd.1 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.71:6802/913345433 v2:172.21.15.71:6803/913345433 exists,up 2f78d6d0-60d6-4135-a871-415a46a14b55', 'osd.2 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.72:6804/3580816619 v2:172.21.15.72:6805/3580816619 exists,up c413b63f-cfa9-481a-abbe-0824b6060779', 'osd.3 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.72:6803/628223927 v2:172.21.15.72:6802/628223927 exists,up 198dfcc4-27f8-41d8-9d0c-33dd50150e68'] 2023-12-15T22:56:37.252 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2023-12-15T22:56:37.359 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-12-15T22:56:37.359 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "mon.a", 2023-12-15T22:56:37.359 INFO:teuthology.orchestra.run.smithi071.stdout: "cluster": "ceph", 2023-12-15T22:56:37.359 INFO:teuthology.orchestra.run.smithi071.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2023-12-15T22:56:37.359 INFO:teuthology.orchestra.run.smithi071.stdout: "admin_socket_mode": "", 2023-12-15T22:56:37.359 INFO:teuthology.orchestra.run.smithi071.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2023-12-15T22:56:37.359 INFO:teuthology.orchestra.run.smithi071.stdout: "auth_client_required": "cephx, none", 2023-12-15T22:56:37.359 INFO:teuthology.orchestra.run.smithi071.stdout: "auth_cluster_required": "cephx", 2023-12-15T22:56:37.359 INFO:teuthology.orchestra.run.smithi071.stdout: "auth_debug": "true", 2023-12-15T22:56:37.359 INFO:teuthology.orchestra.run.smithi071.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2023-12-15T22:56:37.359 INFO:teuthology.orchestra.run.smithi071.stdout: "auth_mon_ticket_ttl": "660.000000", 2023-12-15T22:56:37.360 INFO:teuthology.orchestra.run.smithi071.stdout: "auth_service_required": "cephx", 2023-12-15T22:56:37.360 INFO:teuthology.orchestra.run.smithi071.stdout: "auth_service_ticket_ttl": "240.000000", 2023-12-15T22:56:37.360 INFO:teuthology.orchestra.run.smithi071.stdout: "auth_supported": "cephx", 2023-12-15T22:56:37.360 INFO:teuthology.orchestra.run.smithi071.stdout: "bdev_aio": "true", 2023-12-15T22:56:37.360 INFO:teuthology.orchestra.run.smithi071.stdout: "bdev_aio_max_queue_depth": "1024", 2023-12-15T22:56:37.360 INFO:teuthology.orchestra.run.smithi071.stdout: "bdev_aio_poll_ms": "250", 2023-12-15T22:56:37.360 INFO:teuthology.orchestra.run.smithi071.stdout: "bdev_aio_reap_max": "16", 2023-12-15T22:56:37.360 INFO:teuthology.orchestra.run.smithi071.stdout: "bdev_async_discard": "false", 2023-12-15T22:56:37.360 INFO:teuthology.orchestra.run.smithi071.stdout: "bdev_block_size": "4096", 2023-12-15T22:56:37.360 INFO:teuthology.orchestra.run.smithi071.stdout: "bdev_debug_aio": "false", 2023-12-15T22:56:37.360 INFO:teuthology.orchestra.run.smithi071.stdout: "bdev_debug_aio_log_age": "5.000000", 2023-12-15T22:56:37.360 INFO:teuthology.orchestra.run.smithi071.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2023-12-15T22:56:37.360 INFO:teuthology.orchestra.run.smithi071.stdout: "bdev_debug_inflight_ios": "false", 2023-12-15T22:56:37.360 INFO:teuthology.orchestra.run.smithi071.stdout: "bdev_enable_discard": "false", 2023-12-15T22:56:37.361 INFO:teuthology.orchestra.run.smithi071.stdout: "bdev_flock_retry": "3", 2023-12-15T22:56:37.361 INFO:teuthology.orchestra.run.smithi071.stdout: "bdev_flock_retry_interval": "0.100000", 2023-12-15T22:56:37.361 INFO:teuthology.orchestra.run.smithi071.stdout: "bdev_inject_crash": "0", 2023-12-15T22:56:37.361 INFO:teuthology.orchestra.run.smithi071.stdout: "bdev_inject_crash_flush_delay": "2", 2023-12-15T22:56:37.361 INFO:teuthology.orchestra.run.smithi071.stdout: "bdev_ioring": "false", 2023-12-15T22:56:37.361 INFO:teuthology.orchestra.run.smithi071.stdout: "bdev_ioring_hipri": "false", 2023-12-15T22:56:37.361 INFO:teuthology.orchestra.run.smithi071.stdout: "bdev_ioring_sqthread_poll": "false", 2023-12-15T22:56:37.361 INFO:teuthology.orchestra.run.smithi071.stdout: "bdev_nvme_unbind_from_kernel": "false", 2023-12-15T22:56:37.361 INFO:teuthology.orchestra.run.smithi071.stdout: "bdev_read_buffer_alignment": "4096", 2023-12-15T22:56:37.361 INFO:teuthology.orchestra.run.smithi071.stdout: "bdev_read_preallocated_huge_buffers": "", 2023-12-15T22:56:37.361 INFO:teuthology.orchestra.run.smithi071.stdout: "bdev_type": "", 2023-12-15T22:56:37.361 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_alloc_size": "1048576", 2023-12-15T22:56:37.361 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_allocator": "hybrid", 2023-12-15T22:56:37.361 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_buffered_io": "true", 2023-12-15T22:56:37.362 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_check_for_zeros": "false", 2023-12-15T22:56:37.362 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_check_volume_selector_often": "false", 2023-12-15T22:56:37.362 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_check_volume_selector_on_umount": "false", 2023-12-15T22:56:37.362 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_compact_log_sync": "false", 2023-12-15T22:56:37.362 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2023-12-15T22:56:37.362 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2023-12-15T22:56:37.362 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_log_compact_min_size": "16777216", 2023-12-15T22:56:37.362 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_log_replay_check_allocations": "true", 2023-12-15T22:56:37.362 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_max_log_runway": "4194304", 2023-12-15T22:56:37.362 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_max_prefetch": "1048576", 2023-12-15T22:56:37.362 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_min_flush_size": "524288", 2023-12-15T22:56:37.362 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_min_log_runway": "1048576", 2023-12-15T22:56:37.362 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_replay_recovery": "false", 2023-12-15T22:56:37.362 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_replay_recovery_disable_compact": "false", 2023-12-15T22:56:37.363 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_shared_alloc_size": "65536", 2023-12-15T22:56:37.363 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_sync_write": "false", 2023-12-15T22:56:37.363 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2023-12-15T22:56:37.363 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2023-12-15T22:56:37.363 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2023-12-15T22:56:37.363 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_allocation_from_file": "true", 2023-12-15T22:56:37.363 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_allocator": "hybrid", 2023-12-15T22:56:37.363 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2023-12-15T22:56:37.363 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2023-12-15T22:56:37.363 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2023-12-15T22:56:37.363 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2023-12-15T22:56:37.363 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2023-12-15T22:56:37.363 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_bitmapallocator_span_size": "1024", 2023-12-15T22:56:37.363 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_blobid_prealloc": "10240", 2023-12-15T22:56:37.364 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_block_create": "true", 2023-12-15T22:56:37.364 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_block_db_create": "false", 2023-12-15T22:56:37.364 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_block_db_path": "", 2023-12-15T22:56:37.364 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_block_db_size": "0", 2023-12-15T22:56:37.364 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_block_path": "", 2023-12-15T22:56:37.364 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_block_preallocate_file": "false", 2023-12-15T22:56:37.364 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_block_size": "107374182400", 2023-12-15T22:56:37.364 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_block_wal_create": "false", 2023-12-15T22:56:37.364 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_block_wal_path": "", 2023-12-15T22:56:37.364 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_block_wal_size": "100663296", 2023-12-15T22:56:37.364 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_bluefs": "true", 2023-12-15T22:56:37.364 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2023-12-15T22:56:37.364 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_bluefs_env_mirror": "false", 2023-12-15T22:56:37.364 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_bluefs_max_free": "10737418240", 2023-12-15T22:56:37.365 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2023-12-15T22:56:37.365 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2023-12-15T22:56:37.365 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2023-12-15T22:56:37.365 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2023-12-15T22:56:37.365 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2023-12-15T22:56:37.365 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_autotune": "true", 2023-12-15T22:56:37.365 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_autotune_interval": "5.000000", 2023-12-15T22:56:37.365 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2023-12-15T22:56:37.365 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_kv_ratio": "0.450000", 2023-12-15T22:56:37.365 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_meta_ratio": "0.450000", 2023-12-15T22:56:37.365 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_size": "0", 2023-12-15T22:56:37.365 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_size_hdd": "1073741824", 2023-12-15T22:56:37.365 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_size_ssd": "3221225472", 2023-12-15T22:56:37.365 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_trim_interval": "0.050000", 2023-12-15T22:56:37.366 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2023-12-15T22:56:37.366 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_type": "2q", 2023-12-15T22:56:37.366 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2023-12-15T22:56:37.366 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_clone_cow": "true", 2023-12-15T22:56:37.366 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_compression_algorithm": "snappy", 2023-12-15T22:56:37.366 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_compression_max_blob_size": "0", 2023-12-15T22:56:37.366 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2023-12-15T22:56:37.366 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2023-12-15T22:56:37.366 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_compression_min_blob_size": "0", 2023-12-15T22:56:37.366 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2023-12-15T22:56:37.366 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2023-12-15T22:56:37.366 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_compression_mode": "none", 2023-12-15T22:56:37.366 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_compression_required_ratio": "0.875000", 2023-12-15T22:56:37.366 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_csum_type": "crc32c", 2023-12-15T22:56:37.367 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_debug_enforce_settings": "default", 2023-12-15T22:56:37.367 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_debug_freelist": "false", 2023-12-15T22:56:37.367 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_debug_fsck_abort": "false", 2023-12-15T22:56:37.367 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2023-12-15T22:56:37.367 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2023-12-15T22:56:37.367 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_debug_inject_read_err": "false", 2023-12-15T22:56:37.367 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_debug_legacy_omap": "false", 2023-12-15T22:56:37.367 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_debug_no_reuse_blocks": "false", 2023-12-15T22:56:37.367 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_debug_omit_block_device_write": "false", 2023-12-15T22:56:37.367 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_debug_omit_kv_commit": "false", 2023-12-15T22:56:37.367 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2023-12-15T22:56:37.367 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_debug_prefill": "0.000000", 2023-12-15T22:56:37.367 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_debug_prefragment_max": "1048576", 2023-12-15T22:56:37.367 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_debug_random_read_err": "0.000000", 2023-12-15T22:56:37.367 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2023-12-15T22:56:37.368 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_debug_small_allocations": "0", 2023-12-15T22:56:37.368 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2023-12-15T22:56:37.368 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_default_buffered_read": "true", 2023-12-15T22:56:37.368 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_default_buffered_write": "false", 2023-12-15T22:56:37.368 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_deferred_batch_ops": "0", 2023-12-15T22:56:37.368 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2023-12-15T22:56:37.368 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2023-12-15T22:56:37.368 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_elastic_shared_blobs": "true", 2023-12-15T22:56:37.368 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2023-12-15T22:56:37.368 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_extent_map_shard_max_size": "1200", 2023-12-15T22:56:37.368 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_extent_map_shard_min_size": "150", 2023-12-15T22:56:37.368 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_extent_map_shard_target_size": "500", 2023-12-15T22:56:37.368 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2023-12-15T22:56:37.368 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_fail_eio": "false", 2023-12-15T22:56:37.368 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_freelist_blocks_per_key": "128", 2023-12-15T22:56:37.369 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2023-12-15T22:56:37.369 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2023-12-15T22:56:37.369 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2023-12-15T22:56:37.369 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_fsck_on_mkfs": "true", 2023-12-15T22:56:37.369 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2023-12-15T22:56:37.369 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_fsck_on_mount": "false", 2023-12-15T22:56:37.369 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_fsck_on_mount_deep": "false", 2023-12-15T22:56:37.369 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_fsck_on_umount": "false", 2023-12-15T22:56:37.369 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_fsck_on_umount_deep": "false", 2023-12-15T22:56:37.369 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2023-12-15T22:56:37.369 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_fsck_quick_fix_threads": "2", 2023-12-15T22:56:37.369 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2023-12-15T22:56:37.369 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2023-12-15T22:56:37.370 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_gc_enable_blob_threshold": "0", 2023-12-15T22:56:37.370 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_gc_enable_total_threshold": "0", 2023-12-15T22:56:37.370 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2023-12-15T22:56:37.370 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_ignore_data_csum": "false", 2023-12-15T22:56:37.370 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2023-12-15T22:56:37.370 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_kvbackend": "rocksdb", 2023-12-15T22:56:37.370 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_log_collection_list_age": "60.000000", 2023-12-15T22:56:37.370 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2023-12-15T22:56:37.370 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_log_op_age": "5.000000", 2023-12-15T22:56:37.370 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_max_alloc_size": "0", 2023-12-15T22:56:37.370 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_max_blob_size": "0", 2023-12-15T22:56:37.370 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_max_blob_size_hdd": "65536", 2023-12-15T22:56:37.370 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_max_blob_size_ssd": "65536", 2023-12-15T22:56:37.370 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_max_defer_interval": "3.000000", 2023-12-15T22:56:37.370 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_max_deferred_txc": "32", 2023-12-15T22:56:37.371 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_min_alloc_size": "0", 2023-12-15T22:56:37.371 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_min_alloc_size_hdd": "4096", 2023-12-15T22:56:37.371 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_min_alloc_size_ssd": "4096", 2023-12-15T22:56:37.371 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_nid_prealloc": "1024", 2023-12-15T22:56:37.371 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_prefer_deferred_size": "0", 2023-12-15T22:56:37.371 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2023-12-15T22:56:37.371 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2023-12-15T22:56:37.372 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_qfsck_on_mount": "true", 2023-12-15T22:56:37.372 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_retry_disk_reads": "3", 2023-12-15T22:56:37.372 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_rocksdb_cf": "true", 2023-12-15T22:56:37.372 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2023-12-15T22:56:37.372 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2023-12-15T22:56:37.372 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_rocksdb_options_annex": "", 2023-12-15T22:56:37.372 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_spdk_coremask": "0x1", 2023-12-15T22:56:37.373 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_spdk_io_sleep": "5", 2023-12-15T22:56:37.373 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_spdk_max_io_completion": "0", 2023-12-15T22:56:37.373 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_spdk_mem": "512", 2023-12-15T22:56:37.373 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_sync_submit_transaction": "false", 2023-12-15T22:56:37.373 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_throttle_bytes": "67108864", 2023-12-15T22:56:37.373 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_throttle_cost_per_io": "0", 2023-12-15T22:56:37.373 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2023-12-15T22:56:37.373 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2023-12-15T22:56:37.373 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2023-12-15T22:56:37.373 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_throttle_trace_rate": "0.000000", 2023-12-15T22:56:37.373 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_tracing": "false", 2023-12-15T22:56:37.373 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2023-12-15T22:56:37.373 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2023-12-15T22:56:37.374 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_volume_selection_reserved": "0", 2023-12-15T22:56:37.374 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2023-12-15T22:56:37.374 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2023-12-15T22:56:37.374 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_warn_on_legacy_statfs": "true", 2023-12-15T22:56:37.374 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2023-12-15T22:56:37.374 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2023-12-15T22:56:37.374 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2023-12-15T22:56:37.374 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_zero_block_detection": "false", 2023-12-15T22:56:37.374 INFO:teuthology.orchestra.run.smithi071.stdout: "cephadm_path": "/usr/sbin/cephadm", 2023-12-15T22:56:37.374 INFO:teuthology.orchestra.run.smithi071.stdout: "cephfs_mirror_action_update_interval": "2", 2023-12-15T22:56:37.374 INFO:teuthology.orchestra.run.smithi071.stdout: "cephfs_mirror_directory_scan_interval": "10", 2023-12-15T22:56:37.374 INFO:teuthology.orchestra.run.smithi071.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2023-12-15T22:56:37.374 INFO:teuthology.orchestra.run.smithi071.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2023-12-15T22:56:37.375 INFO:teuthology.orchestra.run.smithi071.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2023-12-15T22:56:37.375 INFO:teuthology.orchestra.run.smithi071.stdout: "cephfs_mirror_mount_timeout": "10", 2023-12-15T22:56:37.375 INFO:teuthology.orchestra.run.smithi071.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2023-12-15T22:56:37.375 INFO:teuthology.orchestra.run.smithi071.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2023-12-15T22:56:37.375 INFO:teuthology.orchestra.run.smithi071.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2023-12-15T22:56:37.375 INFO:teuthology.orchestra.run.smithi071.stdout: "cephsqlite_blocklist_dead_locker": "true", 2023-12-15T22:56:37.375 INFO:teuthology.orchestra.run.smithi071.stdout: "cephsqlite_lock_renewal_interval": "2000", 2023-12-15T22:56:37.375 INFO:teuthology.orchestra.run.smithi071.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2023-12-15T22:56:37.375 INFO:teuthology.orchestra.run.smithi071.stdout: "cephx_cluster_require_signatures": "false", 2023-12-15T22:56:37.375 INFO:teuthology.orchestra.run.smithi071.stdout: "cephx_cluster_require_version": "2", 2023-12-15T22:56:37.375 INFO:teuthology.orchestra.run.smithi071.stdout: "cephx_require_signatures": "false", 2023-12-15T22:56:37.375 INFO:teuthology.orchestra.run.smithi071.stdout: "cephx_require_version": "2", 2023-12-15T22:56:37.375 INFO:teuthology.orchestra.run.smithi071.stdout: "cephx_service_require_signatures": "false", 2023-12-15T22:56:37.375 INFO:teuthology.orchestra.run.smithi071.stdout: "cephx_service_require_version": "2", 2023-12-15T22:56:37.376 INFO:teuthology.orchestra.run.smithi071.stdout: "cephx_sign_messages": "true", 2023-12-15T22:56:37.376 INFO:teuthology.orchestra.run.smithi071.stdout: "chdir": "", 2023-12-15T22:56:37.376 INFO:teuthology.orchestra.run.smithi071.stdout: "client_acl_type": "", 2023-12-15T22:56:37.376 INFO:teuthology.orchestra.run.smithi071.stdout: "client_asio_thread_count": "2", 2023-12-15T22:56:37.376 INFO:teuthology.orchestra.run.smithi071.stdout: "client_cache_mid": "0.750000", 2023-12-15T22:56:37.376 INFO:teuthology.orchestra.run.smithi071.stdout: "client_cache_size": "16384", 2023-12-15T22:56:37.376 INFO:teuthology.orchestra.run.smithi071.stdout: "client_caps_release_delay": "5", 2023-12-15T22:56:37.376 INFO:teuthology.orchestra.run.smithi071.stdout: "client_check_pool_perm": "true", 2023-12-15T22:56:37.376 INFO:teuthology.orchestra.run.smithi071.stdout: "client_collect_and_send_global_metrics": "false", 2023-12-15T22:56:37.376 INFO:teuthology.orchestra.run.smithi071.stdout: "client_debug_force_sync_read": "false", 2023-12-15T22:56:37.376 INFO:teuthology.orchestra.run.smithi071.stdout: "client_debug_getattr_caps": "false", 2023-12-15T22:56:37.376 INFO:teuthology.orchestra.run.smithi071.stdout: "client_debug_inject_tick_delay": "0", 2023-12-15T22:56:37.376 INFO:teuthology.orchestra.run.smithi071.stdout: "client_die_on_failed_dentry_invalidate": "true", 2023-12-15T22:56:37.376 INFO:teuthology.orchestra.run.smithi071.stdout: "client_die_on_failed_remount": "false", 2023-12-15T22:56:37.376 INFO:teuthology.orchestra.run.smithi071.stdout: "client_dirsize_rbytes": "true", 2023-12-15T22:56:37.376 INFO:teuthology.orchestra.run.smithi071.stdout: "client_force_lazyio": "false", 2023-12-15T22:56:37.377 INFO:teuthology.orchestra.run.smithi071.stdout: "client_fs": "", 2023-12-15T22:56:37.377 INFO:teuthology.orchestra.run.smithi071.stdout: "client_inject_fixed_oldest_tid": "false", 2023-12-15T22:56:37.377 INFO:teuthology.orchestra.run.smithi071.stdout: "client_inject_release_failure": "false", 2023-12-15T22:56:37.377 INFO:teuthology.orchestra.run.smithi071.stdout: "client_max_inline_size": "4096", 2023-12-15T22:56:37.377 INFO:teuthology.orchestra.run.smithi071.stdout: "client_max_retries_on_remount_failure": "5", 2023-12-15T22:56:37.377 INFO:teuthology.orchestra.run.smithi071.stdout: "client_mds_namespace": "", 2023-12-15T22:56:37.377 INFO:teuthology.orchestra.run.smithi071.stdout: "client_metadata": "", 2023-12-15T22:56:37.377 INFO:teuthology.orchestra.run.smithi071.stdout: "client_mount_gid": "-1", 2023-12-15T22:56:37.377 INFO:teuthology.orchestra.run.smithi071.stdout: "client_mount_timeout": "300", 2023-12-15T22:56:37.377 INFO:teuthology.orchestra.run.smithi071.stdout: "client_mount_uid": "-1", 2023-12-15T22:56:37.377 INFO:teuthology.orchestra.run.smithi071.stdout: "client_mountpoint": "/", 2023-12-15T22:56:37.377 INFO:teuthology.orchestra.run.smithi071.stdout: "client_notify_timeout": "10", 2023-12-15T22:56:37.377 INFO:teuthology.orchestra.run.smithi071.stdout: "client_oc": "true", 2023-12-15T22:56:37.378 INFO:teuthology.orchestra.run.smithi071.stdout: "client_oc_max_dirty": "104857600", 2023-12-15T22:56:37.378 INFO:teuthology.orchestra.run.smithi071.stdout: "client_oc_max_dirty_age": "5.000000", 2023-12-15T22:56:37.378 INFO:teuthology.orchestra.run.smithi071.stdout: "client_oc_max_objects": "1000", 2023-12-15T22:56:37.378 INFO:teuthology.orchestra.run.smithi071.stdout: "client_oc_size": "209715200", 2023-12-15T22:56:37.378 INFO:teuthology.orchestra.run.smithi071.stdout: "client_oc_target_dirty": "8388608", 2023-12-15T22:56:37.378 INFO:teuthology.orchestra.run.smithi071.stdout: "client_permissions": "true", 2023-12-15T22:56:37.378 INFO:teuthology.orchestra.run.smithi071.stdout: "client_quota": "true", 2023-12-15T22:56:37.378 INFO:teuthology.orchestra.run.smithi071.stdout: "client_quota_df": "true", 2023-12-15T22:56:37.378 INFO:teuthology.orchestra.run.smithi071.stdout: "client_readahead_max_bytes": "0", 2023-12-15T22:56:37.378 INFO:teuthology.orchestra.run.smithi071.stdout: "client_readahead_max_periods": "4", 2023-12-15T22:56:37.378 INFO:teuthology.orchestra.run.smithi071.stdout: "client_readahead_min": "131072", 2023-12-15T22:56:37.378 INFO:teuthology.orchestra.run.smithi071.stdout: "client_reconnect_stale": "false", 2023-12-15T22:56:37.378 INFO:teuthology.orchestra.run.smithi071.stdout: "client_shutdown_timeout": "30", 2023-12-15T22:56:37.378 INFO:teuthology.orchestra.run.smithi071.stdout: "client_snapdir": ".snap", 2023-12-15T22:56:37.378 INFO:teuthology.orchestra.run.smithi071.stdout: "client_tick_interval": "1", 2023-12-15T22:56:37.379 INFO:teuthology.orchestra.run.smithi071.stdout: "client_trace": "", 2023-12-15T22:56:37.379 INFO:teuthology.orchestra.run.smithi071.stdout: "client_try_dentry_invalidate": "false", 2023-12-15T22:56:37.379 INFO:teuthology.orchestra.run.smithi071.stdout: "client_use_faked_inos": "false", 2023-12-15T22:56:37.379 INFO:teuthology.orchestra.run.smithi071.stdout: "client_use_random_mds": "false", 2023-12-15T22:56:37.379 INFO:teuthology.orchestra.run.smithi071.stdout: "clog_to_graylog": "false", 2023-12-15T22:56:37.379 INFO:teuthology.orchestra.run.smithi071.stdout: "clog_to_graylog_host": "127.0.0.1", 2023-12-15T22:56:37.379 INFO:teuthology.orchestra.run.smithi071.stdout: "clog_to_graylog_port": "12201", 2023-12-15T22:56:37.379 INFO:teuthology.orchestra.run.smithi071.stdout: "clog_to_monitors": "default=true", 2023-12-15T22:56:37.379 INFO:teuthology.orchestra.run.smithi071.stdout: "clog_to_syslog": "false", 2023-12-15T22:56:37.379 INFO:teuthology.orchestra.run.smithi071.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2023-12-15T22:56:37.379 INFO:teuthology.orchestra.run.smithi071.stdout: "clog_to_syslog_level": "info", 2023-12-15T22:56:37.379 INFO:teuthology.orchestra.run.smithi071.stdout: "cluster_addr": "-", 2023-12-15T22:56:37.379 INFO:teuthology.orchestra.run.smithi071.stdout: "cluster_network": "", 2023-12-15T22:56:37.379 INFO:teuthology.orchestra.run.smithi071.stdout: "cluster_network_interface": "", 2023-12-15T22:56:37.380 INFO:teuthology.orchestra.run.smithi071.stdout: "compressor_zlib_isal": "false", 2023-12-15T22:56:37.380 INFO:teuthology.orchestra.run.smithi071.stdout: "compressor_zlib_level": "5", 2023-12-15T22:56:37.380 INFO:teuthology.orchestra.run.smithi071.stdout: "compressor_zlib_winsize": "-15", 2023-12-15T22:56:37.380 INFO:teuthology.orchestra.run.smithi071.stdout: "compressor_zstd_level": "1", 2023-12-15T22:56:37.380 INFO:teuthology.orchestra.run.smithi071.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2023-12-15T22:56:37.380 INFO:teuthology.orchestra.run.smithi071.stdout: "crash_dir": "/var/lib/ceph/crash", 2023-12-15T22:56:37.380 INFO:teuthology.orchestra.run.smithi071.stdout: "crimson_alien_op_num_threads": "6", 2023-12-15T22:56:37.380 INFO:teuthology.orchestra.run.smithi071.stdout: "crimson_alien_thread_cpu_cores": "", 2023-12-15T22:56:37.380 INFO:teuthology.orchestra.run.smithi071.stdout: "crimson_osd_obc_lru_size": "10", 2023-12-15T22:56:37.380 INFO:teuthology.orchestra.run.smithi071.stdout: "crimson_osd_scheduler_concurrency": "0", 2023-12-15T22:56:37.380 INFO:teuthology.orchestra.run.smithi071.stdout: "crimson_seastar_smp": "1", 2023-12-15T22:56:37.380 INFO:teuthology.orchestra.run.smithi071.stdout: "crush_location": "", 2023-12-15T22:56:37.380 INFO:teuthology.orchestra.run.smithi071.stdout: "crush_location_hook": "", 2023-12-15T22:56:37.381 INFO:teuthology.orchestra.run.smithi071.stdout: "crush_location_hook_timeout": "10", 2023-12-15T22:56:37.381 INFO:teuthology.orchestra.run.smithi071.stdout: "daemonize": "false", 2023-12-15T22:56:37.381 INFO:teuthology.orchestra.run.smithi071.stdout: "daos_pool": "tank", 2023-12-15T22:56:37.381 INFO:teuthology.orchestra.run.smithi071.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2023-12-15T22:56:37.381 INFO:teuthology.orchestra.run.smithi071.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2023-12-15T22:56:37.381 INFO:teuthology.orchestra.run.smithi071.stdout: "dbstore_db_name_prefix": "dbstore", 2023-12-15T22:56:37.381 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_alienstore": "0/5", 2023-12-15T22:56:37.381 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_allow_any_pool_priority": "false", 2023-12-15T22:56:37.381 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_asok": "1/5", 2023-12-15T22:56:37.381 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_asok_assert_abort": "false", 2023-12-15T22:56:37.381 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_asserts_on_shutdown": "true", 2023-12-15T22:56:37.381 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_auth": "20/20", 2023-12-15T22:56:37.381 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_bdev": "1/3", 2023-12-15T22:56:37.381 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_bluefs": "1/5", 2023-12-15T22:56:37.382 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_bluestore": "1/5", 2023-12-15T22:56:37.382 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_buffer": "0/1", 2023-12-15T22:56:37.382 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_ceph_exporter": "1/5", 2023-12-15T22:56:37.382 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_cephfs_mirror": "0/5", 2023-12-15T22:56:37.382 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_cephsqlite": "0/5", 2023-12-15T22:56:37.382 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_client": "0/5", 2023-12-15T22:56:37.382 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_compressor": "1/5", 2023-12-15T22:56:37.382 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_context": "0/1", 2023-12-15T22:56:37.382 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_crimson_interrupt": "0/5", 2023-12-15T22:56:37.382 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_crush": "1/1", 2023-12-15T22:56:37.382 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_crypto": "1/5", 2023-12-15T22:56:37.382 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_cyanstore": "0/5", 2023-12-15T22:56:37.382 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_deliberately_leak_memory": "false", 2023-12-15T22:56:37.382 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_disable_randomized_ping": "false", 2023-12-15T22:56:37.383 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_dpdk": "1/5", 2023-12-15T22:56:37.383 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_eventtrace": "1/5", 2023-12-15T22:56:37.383 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_filer": "0/1", 2023-12-15T22:56:37.383 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_filestore": "1/3", 2023-12-15T22:56:37.383 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_finisher": "1/1", 2023-12-15T22:56:37.383 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_fuse": "1/5", 2023-12-15T22:56:37.383 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_heartbeat_testing_span": "0", 2023-12-15T22:56:37.383 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_heartbeatmap": "1/5", 2023-12-15T22:56:37.383 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_immutable_obj_cache": "0/5", 2023-12-15T22:56:37.383 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_javaclient": "1/5", 2023-12-15T22:56:37.383 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_journal": "1/3", 2023-12-15T22:56:37.383 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_journaler": "0/5", 2023-12-15T22:56:37.383 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_kstore": "1/5", 2023-12-15T22:56:37.383 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_lockdep": "0/1", 2023-12-15T22:56:37.383 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_mclock": "1/5", 2023-12-15T22:56:37.383 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_mds": "1/5", 2023-12-15T22:56:37.384 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_mds_balancer": "1/5", 2023-12-15T22:56:37.384 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_mds_locker": "1/5", 2023-12-15T22:56:37.384 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_mds_log": "1/5", 2023-12-15T22:56:37.384 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_mds_log_expire": "1/5", 2023-12-15T22:56:37.384 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_mds_migrator": "1/5", 2023-12-15T22:56:37.384 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_memstore": "1/5", 2023-12-15T22:56:37.384 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_mgr": "2/5", 2023-12-15T22:56:37.384 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_mgrc": "1/5", 2023-12-15T22:56:37.384 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_mon": "20/20", 2023-12-15T22:56:37.384 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_monc": "0/10", 2023-12-15T22:56:37.384 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_ms": "1/1", 2023-12-15T22:56:37.384 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_none": "0/5", 2023-12-15T22:56:37.384 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_objclass": "0/5", 2023-12-15T22:56:37.385 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_objectcacher": "0/5", 2023-12-15T22:56:37.385 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_objecter": "0/1", 2023-12-15T22:56:37.385 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_optracker": "0/5", 2023-12-15T22:56:37.385 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_osd": "1/5", 2023-12-15T22:56:37.385 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_paxos": "20/20", 2023-12-15T22:56:37.385 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_perfcounter": "1/5", 2023-12-15T22:56:37.385 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_prioritycache": "1/5", 2023-12-15T22:56:37.385 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_rados": "0/5", 2023-12-15T22:56:37.385 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_rbd": "0/5", 2023-12-15T22:56:37.385 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_rbd_mirror": "0/5", 2023-12-15T22:56:37.385 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_rbd_pwl": "0/5", 2023-12-15T22:56:37.385 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_rbd_replay": "0/5", 2023-12-15T22:56:37.385 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_refs": "0/0", 2023-12-15T22:56:37.385 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_reserver": "1/1", 2023-12-15T22:56:37.385 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_rgw": "1/5", 2023-12-15T22:56:37.386 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_rgw_access": "1/5", 2023-12-15T22:56:37.386 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_rgw_datacache": "1/5", 2023-12-15T22:56:37.386 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_rgw_dbstore": "1/5", 2023-12-15T22:56:37.386 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_rgw_flight": "1/5", 2023-12-15T22:56:37.386 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_rgw_sync": "1/5", 2023-12-15T22:56:37.386 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_rocksdb": "4/5", 2023-12-15T22:56:37.386 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_seastore": "0/5", 2023-12-15T22:56:37.386 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_seastore_backref": "0/5", 2023-12-15T22:56:37.386 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_seastore_cache": "0/5", 2023-12-15T22:56:37.386 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_seastore_cleaner": "0/5", 2023-12-15T22:56:37.386 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_seastore_device": "0/5", 2023-12-15T22:56:37.386 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_seastore_epm": "0/5", 2023-12-15T22:56:37.386 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_seastore_fixedkv_tree": "0/5", 2023-12-15T22:56:37.386 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_seastore_journal": "0/5", 2023-12-15T22:56:37.386 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_seastore_lba": "0/5", 2023-12-15T22:56:37.387 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_seastore_odata": "0/5", 2023-12-15T22:56:37.387 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_seastore_omap": "0/5", 2023-12-15T22:56:37.387 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_seastore_onode": "0/5", 2023-12-15T22:56:37.387 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_seastore_t": "0/5", 2023-12-15T22:56:37.387 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_seastore_tm": "0/5", 2023-12-15T22:56:37.387 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_striper": "0/1", 2023-12-15T22:56:37.387 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_test": "0/5", 2023-12-15T22:56:37.387 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_throttle": "1/1", 2023-12-15T22:56:37.387 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_timer": "0/1", 2023-12-15T22:56:37.387 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_tp": "0/5", 2023-12-15T22:56:37.387 INFO:teuthology.orchestra.run.smithi071.stdout: "debug_trace": "1/5", 2023-12-15T22:56:37.387 INFO:teuthology.orchestra.run.smithi071.stdout: "defer_client_eviction_on_laggy_osds": "true", 2023-12-15T22:56:37.387 INFO:teuthology.orchestra.run.smithi071.stdout: "device_failure_prediction_mode": "none", 2023-12-15T22:56:37.387 INFO:teuthology.orchestra.run.smithi071.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "crimson", 2023-12-15T22:56:37.388 INFO:teuthology.orchestra.run.smithi071.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2023-12-15T22:56:37.388 INFO:teuthology.orchestra.run.smithi071.stdout: "err_to_graylog": "false", 2023-12-15T22:56:37.388 INFO:teuthology.orchestra.run.smithi071.stdout: "err_to_journald": "false", 2023-12-15T22:56:37.388 INFO:teuthology.orchestra.run.smithi071.stdout: "err_to_stderr": "true", 2023-12-15T22:56:37.388 INFO:teuthology.orchestra.run.smithi071.stdout: "err_to_syslog": "false", 2023-12-15T22:56:37.388 INFO:teuthology.orchestra.run.smithi071.stdout: "event_tracing": "false", 2023-12-15T22:56:37.388 INFO:teuthology.orchestra.run.smithi071.stdout: "exporter_addr": "0.0.0.0", 2023-12-15T22:56:37.388 INFO:teuthology.orchestra.run.smithi071.stdout: "exporter_http_port": "9926", 2023-12-15T22:56:37.388 INFO:teuthology.orchestra.run.smithi071.stdout: "exporter_prio_limit": "5", 2023-12-15T22:56:37.388 INFO:teuthology.orchestra.run.smithi071.stdout: "exporter_sock_dir": "/var/run/ceph/", 2023-12-15T22:56:37.388 INFO:teuthology.orchestra.run.smithi071.stdout: "exporter_sort_metrics": "true", 2023-12-15T22:56:37.388 INFO:teuthology.orchestra.run.smithi071.stdout: "exporter_stats_period": "5", 2023-12-15T22:56:37.388 INFO:teuthology.orchestra.run.smithi071.stdout: "fake_statfs_for_testing": "0", 2023-12-15T22:56:37.388 INFO:teuthology.orchestra.run.smithi071.stdout: "fatal_signal_handlers": "true", 2023-12-15T22:56:37.389 INFO:teuthology.orchestra.run.smithi071.stdout: "filer_max_purge_ops": "10", 2023-12-15T22:56:37.389 INFO:teuthology.orchestra.run.smithi071.stdout: "filer_max_truncate_ops": "128", 2023-12-15T22:56:37.389 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_apply_finisher_threads": "1", 2023-12-15T22:56:37.389 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_blackhole": "false", 2023-12-15T22:56:37.389 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_btrfs_clone_range": "true", 2023-12-15T22:56:37.389 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_btrfs_snap": "true", 2023-12-15T22:56:37.389 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_caller_concurrency": "10", 2023-12-15T22:56:37.389 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_collect_device_partition_information": "true", 2023-12-15T22:56:37.389 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_commit_timeout": "600.000000", 2023-12-15T22:56:37.389 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_debug_inject_read_err": "false", 2023-12-15T22:56:37.389 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_debug_omap_check": "false", 2023-12-15T22:56:37.389 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_debug_random_read_err": "0.000000", 2023-12-15T22:56:37.389 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_debug_verify_split": "false", 2023-12-15T22:56:37.389 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_dump_file": "", 2023-12-15T22:56:37.390 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2023-12-15T22:56:37.390 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_expected_throughput_ops": "200.000000", 2023-12-15T22:56:37.390 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_fadvise": "true", 2023-12-15T22:56:37.390 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_fail_eio": "true", 2023-12-15T22:56:37.390 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_fd_cache_shards": "16", 2023-12-15T22:56:37.390 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_fd_cache_size": "128", 2023-12-15T22:56:37.390 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_fiemap": "false", 2023-12-15T22:56:37.390 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_fiemap_threshold": "4096", 2023-12-15T22:56:37.390 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_fsync_flushes_journal_data": "false", 2023-12-15T22:56:37.390 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_index_retry_probability": "0.000000", 2023-12-15T22:56:37.390 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_inject_stall": "0", 2023-12-15T22:56:37.390 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_journal_parallel": "false", 2023-12-15T22:56:37.390 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_journal_trailing": "false", 2023-12-15T22:56:37.390 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_journal_writeahead": "false", 2023-12-15T22:56:37.390 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_kill_at": "0", 2023-12-15T22:56:37.391 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_max_alloc_hint_size": "1048576", 2023-12-15T22:56:37.391 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_max_inline_xattr_size": "0", 2023-12-15T22:56:37.391 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2023-12-15T22:56:37.391 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_max_inline_xattr_size_other": "512", 2023-12-15T22:56:37.391 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2023-12-15T22:56:37.391 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_max_inline_xattrs": "0", 2023-12-15T22:56:37.391 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2023-12-15T22:56:37.391 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_max_inline_xattrs_other": "2", 2023-12-15T22:56:37.391 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_max_inline_xattrs_xfs": "10", 2023-12-15T22:56:37.391 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_max_sync_interval": "5.000000", 2023-12-15T22:56:37.391 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_max_xattr_value_size": "0", 2023-12-15T22:56:37.391 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2023-12-15T22:56:37.391 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_max_xattr_value_size_other": "1024", 2023-12-15T22:56:37.391 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2023-12-15T22:56:37.391 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_merge_threshold": "-10", 2023-12-15T22:56:37.392 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_min_sync_interval": "0.010000", 2023-12-15T22:56:37.392 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_odsync_write": "false", 2023-12-15T22:56:37.392 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_omap_backend": "rocksdb", 2023-12-15T22:56:37.392 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_omap_backend_path": "", 2023-12-15T22:56:37.392 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_omap_header_cache_size": "1024", 2023-12-15T22:56:37.392 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_ondisk_finisher_threads": "1", 2023-12-15T22:56:37.392 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_op_thread_suicide_timeout": "180", 2023-12-15T22:56:37.392 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_op_thread_timeout": "60", 2023-12-15T22:56:37.392 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_op_threads": "2", 2023-12-15T22:56:37.392 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_punch_hole": "false", 2023-12-15T22:56:37.392 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2023-12-15T22:56:37.392 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2023-12-15T22:56:37.392 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2023-12-15T22:56:37.392 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_queue_high_threshhold": "0.900000", 2023-12-15T22:56:37.393 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_queue_low_threshhold": "0.300000", 2023-12-15T22:56:37.393 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_queue_max_bytes": "104857600", 2023-12-15T22:56:37.393 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2023-12-15T22:56:37.393 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2023-12-15T22:56:37.393 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2023-12-15T22:56:37.393 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_queue_max_ops": "50", 2023-12-15T22:56:37.393 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2023-12-15T22:56:37.393 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_seek_data_hole": "false", 2023-12-15T22:56:37.393 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_sloppy_crc": "false", 2023-12-15T22:56:37.393 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_sloppy_crc_block_size": "65536", 2023-12-15T22:56:37.393 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_splice": "false", 2023-12-15T22:56:37.393 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_split_multiple": "2", 2023-12-15T22:56:37.393 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_split_rand_factor": "20", 2023-12-15T22:56:37.393 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_update_to": "1000", 2023-12-15T22:56:37.394 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2023-12-15T22:56:37.394 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2023-12-15T22:56:37.394 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2023-12-15T22:56:37.394 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2023-12-15T22:56:37.394 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2023-12-15T22:56:37.394 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2023-12-15T22:56:37.394 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_wbthrottle_enable": "true", 2023-12-15T22:56:37.394 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2023-12-15T22:56:37.394 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2023-12-15T22:56:37.394 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2023-12-15T22:56:37.394 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2023-12-15T22:56:37.394 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2023-12-15T22:56:37.394 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2023-12-15T22:56:37.394 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_xfs_extsize": "false", 2023-12-15T22:56:37.395 INFO:teuthology.orchestra.run.smithi071.stdout: "filestore_zfs_snap": "false", 2023-12-15T22:56:37.395 INFO:teuthology.orchestra.run.smithi071.stdout: "fio_dir": "/tmp/fio", 2023-12-15T22:56:37.395 INFO:teuthology.orchestra.run.smithi071.stdout: "fsid": "6b8c5e6b-dc21-4b86-85c9-629eae1ef431", 2023-12-15T22:56:37.395 INFO:teuthology.orchestra.run.smithi071.stdout: "fuse_allow_other": "true", 2023-12-15T22:56:37.395 INFO:teuthology.orchestra.run.smithi071.stdout: "fuse_atomic_o_trunc": "true", 2023-12-15T22:56:37.395 INFO:teuthology.orchestra.run.smithi071.stdout: "fuse_big_writes": "true", 2023-12-15T22:56:37.395 INFO:teuthology.orchestra.run.smithi071.stdout: "fuse_debug": "false", 2023-12-15T22:56:37.395 INFO:teuthology.orchestra.run.smithi071.stdout: "fuse_default_permissions": "false", 2023-12-15T22:56:37.395 INFO:teuthology.orchestra.run.smithi071.stdout: "fuse_disable_pagecache": "false", 2023-12-15T22:56:37.395 INFO:teuthology.orchestra.run.smithi071.stdout: "fuse_max_write": "0", 2023-12-15T22:56:37.395 INFO:teuthology.orchestra.run.smithi071.stdout: "fuse_multithreaded": "true", 2023-12-15T22:56:37.395 INFO:teuthology.orchestra.run.smithi071.stdout: "fuse_require_active_mds": "true", 2023-12-15T22:56:37.395 INFO:teuthology.orchestra.run.smithi071.stdout: "fuse_set_user_groups": "true", 2023-12-15T22:56:37.395 INFO:teuthology.orchestra.run.smithi071.stdout: "fuse_splice_move": "true", 2023-12-15T22:56:37.396 INFO:teuthology.orchestra.run.smithi071.stdout: "fuse_splice_read": "true", 2023-12-15T22:56:37.396 INFO:teuthology.orchestra.run.smithi071.stdout: "fuse_splice_write": "true", 2023-12-15T22:56:37.396 INFO:teuthology.orchestra.run.smithi071.stdout: "fuse_syncfs_on_mksnap": "true", 2023-12-15T22:56:37.396 INFO:teuthology.orchestra.run.smithi071.stdout: "fuse_use_invalidate_cb": "true", 2023-12-15T22:56:37.396 INFO:teuthology.orchestra.run.smithi071.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2023-12-15T22:56:37.396 INFO:teuthology.orchestra.run.smithi071.stdout: "gss_target_name": "ceph", 2023-12-15T22:56:37.396 INFO:teuthology.orchestra.run.smithi071.stdout: "heartbeat_file": "", 2023-12-15T22:56:37.396 INFO:teuthology.orchestra.run.smithi071.stdout: "heartbeat_inject_failure": "0", 2023-12-15T22:56:37.396 INFO:teuthology.orchestra.run.smithi071.stdout: "heartbeat_interval": "5", 2023-12-15T22:56:37.396 INFO:teuthology.orchestra.run.smithi071.stdout: "host": "smithi071", 2023-12-15T22:56:37.396 INFO:teuthology.orchestra.run.smithi071.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2023-12-15T22:56:37.396 INFO:teuthology.orchestra.run.smithi071.stdout: "immutable_object_cache_max_inflight_ops": "128", 2023-12-15T22:56:37.396 INFO:teuthology.orchestra.run.smithi071.stdout: "immutable_object_cache_max_size": "1073741824", 2023-12-15T22:56:37.396 INFO:teuthology.orchestra.run.smithi071.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2023-12-15T22:56:37.396 INFO:teuthology.orchestra.run.smithi071.stdout: "immutable_object_cache_qos_bps_burst": "0", 2023-12-15T22:56:37.397 INFO:teuthology.orchestra.run.smithi071.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2023-12-15T22:56:37.397 INFO:teuthology.orchestra.run.smithi071.stdout: "immutable_object_cache_qos_bps_limit": "0", 2023-12-15T22:56:37.397 INFO:teuthology.orchestra.run.smithi071.stdout: "immutable_object_cache_qos_iops_burst": "0", 2023-12-15T22:56:37.397 INFO:teuthology.orchestra.run.smithi071.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2023-12-15T22:56:37.397 INFO:teuthology.orchestra.run.smithi071.stdout: "immutable_object_cache_qos_iops_limit": "0", 2023-12-15T22:56:37.397 INFO:teuthology.orchestra.run.smithi071.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2023-12-15T22:56:37.397 INFO:teuthology.orchestra.run.smithi071.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2023-12-15T22:56:37.397 INFO:teuthology.orchestra.run.smithi071.stdout: "immutable_object_cache_watermark": "0.900000", 2023-12-15T22:56:37.397 INFO:teuthology.orchestra.run.smithi071.stdout: "inject_early_sigterm": "false", 2023-12-15T22:56:37.397 INFO:teuthology.orchestra.run.smithi071.stdout: "jaeger_agent_port": "6799", 2023-12-15T22:56:37.397 INFO:teuthology.orchestra.run.smithi071.stdout: "jaeger_tracing_enable": "false", 2023-12-15T22:56:37.397 INFO:teuthology.orchestra.run.smithi071.stdout: "journal_aio": "true", 2023-12-15T22:56:37.397 INFO:teuthology.orchestra.run.smithi071.stdout: "journal_align_min_size": "65536", 2023-12-15T22:56:37.397 INFO:teuthology.orchestra.run.smithi071.stdout: "journal_block_align": "true", 2023-12-15T22:56:37.398 INFO:teuthology.orchestra.run.smithi071.stdout: "journal_block_size": "4096", 2023-12-15T22:56:37.398 INFO:teuthology.orchestra.run.smithi071.stdout: "journal_dio": "true", 2023-12-15T22:56:37.398 INFO:teuthology.orchestra.run.smithi071.stdout: "journal_discard": "false", 2023-12-15T22:56:37.398 INFO:teuthology.orchestra.run.smithi071.stdout: "journal_force_aio": "false", 2023-12-15T22:56:37.398 INFO:teuthology.orchestra.run.smithi071.stdout: "journal_ignore_corruption": "false", 2023-12-15T22:56:37.398 INFO:teuthology.orchestra.run.smithi071.stdout: "journal_max_write_bytes": "10485760", 2023-12-15T22:56:37.398 INFO:teuthology.orchestra.run.smithi071.stdout: "journal_max_write_entries": "100", 2023-12-15T22:56:37.398 INFO:teuthology.orchestra.run.smithi071.stdout: "journal_replay_from": "0", 2023-12-15T22:56:37.398 INFO:teuthology.orchestra.run.smithi071.stdout: "journal_throttle_high_multiple": "0.000000", 2023-12-15T22:56:37.398 INFO:teuthology.orchestra.run.smithi071.stdout: "journal_throttle_high_threshhold": "0.900000", 2023-12-15T22:56:37.398 INFO:teuthology.orchestra.run.smithi071.stdout: "journal_throttle_low_threshhold": "0.600000", 2023-12-15T22:56:37.398 INFO:teuthology.orchestra.run.smithi071.stdout: "journal_throttle_max_multiple": "0.000000", 2023-12-15T22:56:37.398 INFO:teuthology.orchestra.run.smithi071.stdout: "journal_write_header_frequency": "0", 2023-12-15T22:56:37.398 INFO:teuthology.orchestra.run.smithi071.stdout: "journal_zero_on_create": "false", 2023-12-15T22:56:37.398 INFO:teuthology.orchestra.run.smithi071.stdout: "journaler_prefetch_periods": "10", 2023-12-15T22:56:37.399 INFO:teuthology.orchestra.run.smithi071.stdout: "journaler_prezero_periods": "5", 2023-12-15T22:56:37.399 INFO:teuthology.orchestra.run.smithi071.stdout: "journaler_write_head_interval": "15", 2023-12-15T22:56:37.399 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "", 2023-12-15T22:56:37.399 INFO:teuthology.orchestra.run.smithi071.stdout: "keyfile": "", 2023-12-15T22:56:37.399 INFO:teuthology.orchestra.run.smithi071.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2023-12-15T22:56:37.399 INFO:teuthology.orchestra.run.smithi071.stdout: "kstore_backend": "rocksdb", 2023-12-15T22:56:37.399 INFO:teuthology.orchestra.run.smithi071.stdout: "kstore_default_stripe_size": "65536", 2023-12-15T22:56:37.399 INFO:teuthology.orchestra.run.smithi071.stdout: "kstore_fsck_on_mount": "false", 2023-12-15T22:56:37.399 INFO:teuthology.orchestra.run.smithi071.stdout: "kstore_fsck_on_mount_deep": "true", 2023-12-15T22:56:37.399 INFO:teuthology.orchestra.run.smithi071.stdout: "kstore_max_bytes": "67108864", 2023-12-15T22:56:37.399 INFO:teuthology.orchestra.run.smithi071.stdout: "kstore_max_ops": "512", 2023-12-15T22:56:37.399 INFO:teuthology.orchestra.run.smithi071.stdout: "kstore_nid_prealloc": "1024", 2023-12-15T22:56:37.399 INFO:teuthology.orchestra.run.smithi071.stdout: "kstore_onode_map_size": "1024", 2023-12-15T22:56:37.399 INFO:teuthology.orchestra.run.smithi071.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2023-12-15T22:56:37.400 INFO:teuthology.orchestra.run.smithi071.stdout: "kstore_sync_submit_transaction": "false", 2023-12-15T22:56:37.400 INFO:teuthology.orchestra.run.smithi071.stdout: "kstore_sync_transaction": "false", 2023-12-15T22:56:37.400 INFO:teuthology.orchestra.run.smithi071.stdout: "librados_thread_count": "2", 2023-12-15T22:56:37.400 INFO:teuthology.orchestra.run.smithi071.stdout: "lockdep": "false", 2023-12-15T22:56:37.400 INFO:teuthology.orchestra.run.smithi071.stdout: "lockdep_force_backtrace": "false", 2023-12-15T22:56:37.400 INFO:teuthology.orchestra.run.smithi071.stdout: "log_coarse_timestamps": "true", 2023-12-15T22:56:37.400 INFO:teuthology.orchestra.run.smithi071.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2023-12-15T22:56:37.400 INFO:teuthology.orchestra.run.smithi071.stdout: "log_flush_on_exit": "false", 2023-12-15T22:56:37.400 INFO:teuthology.orchestra.run.smithi071.stdout: "log_graylog_host": "127.0.0.1", 2023-12-15T22:56:37.400 INFO:teuthology.orchestra.run.smithi071.stdout: "log_graylog_port": "12201", 2023-12-15T22:56:37.400 INFO:teuthology.orchestra.run.smithi071.stdout: "log_max_new": "1000", 2023-12-15T22:56:37.400 INFO:teuthology.orchestra.run.smithi071.stdout: "log_max_recent": "10000", 2023-12-15T22:56:37.400 INFO:teuthology.orchestra.run.smithi071.stdout: "log_stderr_prefix": "", 2023-12-15T22:56:37.400 INFO:teuthology.orchestra.run.smithi071.stdout: "log_stop_at_utilization": "0.970000", 2023-12-15T22:56:37.400 INFO:teuthology.orchestra.run.smithi071.stdout: "log_to_file": "true", 2023-12-15T22:56:37.401 INFO:teuthology.orchestra.run.smithi071.stdout: "log_to_graylog": "false", 2023-12-15T22:56:37.401 INFO:teuthology.orchestra.run.smithi071.stdout: "log_to_journald": "false", 2023-12-15T22:56:37.401 INFO:teuthology.orchestra.run.smithi071.stdout: "log_to_stderr": "false", 2023-12-15T22:56:37.401 INFO:teuthology.orchestra.run.smithi071.stdout: "log_to_syslog": "false", 2023-12-15T22:56:37.401 INFO:teuthology.orchestra.run.smithi071.stdout: "mandatory_topic_permissions": "false", 2023-12-15T22:56:37.401 INFO:teuthology.orchestra.run.smithi071.stdout: "max_rotating_auth_attempts": "10", 2023-12-15T22:56:37.401 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2023-12-15T22:56:37.401 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_action_on_write_error": "1", 2023-12-15T22:56:37.401 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_alternate_name_max": "8192", 2023-12-15T22:56:37.401 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_asio_thread_count": "2", 2023-12-15T22:56:37.401 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_bal_export_pin": "true", 2023-12-15T22:56:37.401 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_bal_fragment_dirs": "true", 2023-12-15T22:56:37.401 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2023-12-15T22:56:37.401 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_bal_fragment_interval": "5", 2023-12-15T22:56:37.402 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_bal_fragment_size_max": "100000", 2023-12-15T22:56:37.402 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_bal_idle_threshold": "0.000000", 2023-12-15T22:56:37.402 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_bal_interval": "10", 2023-12-15T22:56:37.402 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_bal_max": "-1", 2023-12-15T22:56:37.402 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_bal_max_until": "-1", 2023-12-15T22:56:37.402 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_bal_merge_size": "50", 2023-12-15T22:56:37.402 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_bal_midchunk": "0.300000", 2023-12-15T22:56:37.402 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_bal_min_rebalance": "0.100000", 2023-12-15T22:56:37.402 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_bal_min_start": "0.200000", 2023-12-15T22:56:37.402 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_bal_minchunk": "0.001000", 2023-12-15T22:56:37.402 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_bal_mode": "0", 2023-12-15T22:56:37.402 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_bal_need_max": "1.200000", 2023-12-15T22:56:37.402 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_bal_need_min": "0.800000", 2023-12-15T22:56:37.402 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_bal_overload_epochs": "2", 2023-12-15T22:56:37.403 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_bal_replicate_threshold": "8000.000000", 2023-12-15T22:56:37.403 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_bal_sample_interval": "3.000000", 2023-12-15T22:56:37.403 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_bal_split_bits": "3", 2023-12-15T22:56:37.403 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_bal_split_rd": "25000.000000", 2023-12-15T22:56:37.403 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_bal_split_size": "10000", 2023-12-15T22:56:37.403 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_bal_split_wr": "10000.000000", 2023-12-15T22:56:37.403 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_bal_target_decay": "10.000000", 2023-12-15T22:56:37.403 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2023-12-15T22:56:37.403 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_beacon_grace": "15.000000", 2023-12-15T22:56:37.403 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_beacon_interval": "4.000000", 2023-12-15T22:56:37.403 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_beacon_mon_down_grace": "60", 2023-12-15T22:56:37.403 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_cache_memory_limit": "4294967296", 2023-12-15T22:56:37.403 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_cache_mid": "0.700000", 2023-12-15T22:56:37.403 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_cache_release_free_interval": "10", 2023-12-15T22:56:37.403 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_cache_reservation": "0.050000", 2023-12-15T22:56:37.404 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_cache_trim_decay_rate": "1.000000", 2023-12-15T22:56:37.404 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_cache_trim_interval": "1", 2023-12-15T22:56:37.404 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_cache_trim_threshold": "262144", 2023-12-15T22:56:37.404 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2023-12-15T22:56:37.404 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2023-12-15T22:56:37.404 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_client_delegate_inos_pct": "50", 2023-12-15T22:56:37.404 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_client_prealloc_inos": "1000", 2023-12-15T22:56:37.404 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2023-12-15T22:56:37.404 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_connect_bootstrapping": "false", 2023-12-15T22:56:37.404 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_damage_table_max_entries": "10000", 2023-12-15T22:56:37.404 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2023-12-15T22:56:37.404 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_debug_auth_pins": "false", 2023-12-15T22:56:37.404 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_debug_frag": "false", 2023-12-15T22:56:37.404 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_debug_scatterstat": "false", 2023-12-15T22:56:37.405 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_debug_subtrees": "false", 2023-12-15T22:56:37.405 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_decay_halflife": "5.000000", 2023-12-15T22:56:37.405 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_default_dir_hash": "2", 2023-12-15T22:56:37.405 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_defer_session_stale": "true", 2023-12-15T22:56:37.405 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_deny_all_reconnect": "false", 2023-12-15T22:56:37.405 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_dir_keys_per_op": "16384", 2023-12-15T22:56:37.405 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_dir_max_commit_size": "10", 2023-12-15T22:56:37.405 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_dir_max_entries": "0", 2023-12-15T22:56:37.405 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_dir_prefetch": "true", 2023-12-15T22:56:37.405 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_dirstat_min_interval": "1.000000", 2023-12-15T22:56:37.405 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_dump_cache_after_rejoin": "false", 2023-12-15T22:56:37.405 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_dump_cache_on_map": "false", 2023-12-15T22:56:37.405 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_dump_cache_threshold_file": "0", 2023-12-15T22:56:37.405 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2023-12-15T22:56:37.406 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_early_reply": "true", 2023-12-15T22:56:37.406 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_enable_op_tracker": "true", 2023-12-15T22:56:37.406 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_enforce_unique_name": "true", 2023-12-15T22:56:37.406 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_export_ephemeral_distributed": "true", 2023-12-15T22:56:37.406 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2023-12-15T22:56:37.406 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_export_ephemeral_random": "true", 2023-12-15T22:56:37.406 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_export_ephemeral_random_max": "0.010000", 2023-12-15T22:56:37.406 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_extraordinary_events_dump_interval": "0", 2023-12-15T22:56:37.406 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_forward_all_requests_to_auth": "false", 2023-12-15T22:56:37.406 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_freeze_tree_timeout": "30.000000", 2023-12-15T22:56:37.406 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_fscrypt_last_block_max_size": "4096", 2023-12-15T22:56:37.406 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_go_bad_corrupt_dentry": "true", 2023-12-15T22:56:37.406 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2023-12-15T22:56:37.406 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_health_cache_threshold": "1.500000", 2023-12-15T22:56:37.406 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_health_summarize_threshold": "10", 2023-12-15T22:56:37.407 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_heartbeat_grace": "15.000000", 2023-12-15T22:56:37.407 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_heartbeat_reset_grace": "1000", 2023-12-15T22:56:37.407 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_inject_health_dummy": "false", 2023-12-15T22:56:37.407 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2023-12-15T22:56:37.407 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_inject_migrator_session_race": "false", 2023-12-15T22:56:37.407 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2023-12-15T22:56:37.407 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_inject_skip_replaying_inotable": "false", 2023-12-15T22:56:37.407 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2023-12-15T22:56:37.407 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_join_fs": "", 2023-12-15T22:56:37.407 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_journal_format": "1", 2023-12-15T22:56:37.407 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_kill_create_at": "0", 2023-12-15T22:56:37.407 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_kill_export_at": "0", 2023-12-15T22:56:37.407 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_kill_import_at": "0", 2023-12-15T22:56:37.407 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_kill_journal_at": "0", 2023-12-15T22:56:37.407 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_kill_journal_expire_at": "0", 2023-12-15T22:56:37.408 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_kill_journal_replay_at": "0", 2023-12-15T22:56:37.408 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_kill_link_at": "0", 2023-12-15T22:56:37.408 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_kill_mdstable_at": "0", 2023-12-15T22:56:37.408 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_kill_openc_at": "0", 2023-12-15T22:56:37.408 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_kill_rename_at": "0", 2023-12-15T22:56:37.408 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_kill_shutdown_at": "0", 2023-12-15T22:56:37.408 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_kill_skip_replaying_inotable": "false", 2023-12-15T22:56:37.408 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_log_event_large_threshold": "524288", 2023-12-15T22:56:37.408 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_log_events_per_segment": "1024", 2023-12-15T22:56:37.408 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_log_major_segment_event_ratio": "12", 2023-12-15T22:56:37.408 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_log_max_events": "-1", 2023-12-15T22:56:37.408 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_log_max_segments": "128", 2023-12-15T22:56:37.408 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_log_pause": "false", 2023-12-15T22:56:37.408 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_log_segment_size": "0", 2023-12-15T22:56:37.408 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_log_skip_corrupt_events": "false", 2023-12-15T22:56:37.409 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_log_skip_unbounded_events": "false", 2023-12-15T22:56:37.409 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_log_warn_factor": "2.000000", 2023-12-15T22:56:37.409 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_max_caps_per_client": "1048576", 2023-12-15T22:56:37.409 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_max_completed_flushes": "100000", 2023-12-15T22:56:37.409 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_max_completed_requests": "100000", 2023-12-15T22:56:37.409 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_max_export_size": "20971520", 2023-12-15T22:56:37.409 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_max_file_recover": "32", 2023-12-15T22:56:37.409 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_max_purge_files": "64", 2023-12-15T22:56:37.409 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_max_purge_ops": "8192", 2023-12-15T22:56:37.409 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2023-12-15T22:56:37.409 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_max_scrub_ops_in_progress": "5", 2023-12-15T22:56:37.409 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_max_snaps_per_dir": "100", 2023-12-15T22:56:37.409 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_metrics_update_interval": "2", 2023-12-15T22:56:37.409 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_min_caps_per_client": "100", 2023-12-15T22:56:37.410 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_min_caps_working_set": "10000", 2023-12-15T22:56:37.410 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_mon_shutdown_timeout": "5.000000", 2023-12-15T22:56:37.410 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_numa_node": "-1", 2023-12-15T22:56:37.410 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_oft_prefetch_dirfrags": "false", 2023-12-15T22:56:37.410 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_op_complaint_time": "30.000000", 2023-12-15T22:56:37.410 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_op_history_duration": "600", 2023-12-15T22:56:37.410 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_op_history_size": "20", 2023-12-15T22:56:37.410 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_op_history_slow_op_size": "20", 2023-12-15T22:56:37.410 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_op_history_slow_op_threshold": "10", 2023-12-15T22:56:37.410 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_op_log_threshold": "5", 2023-12-15T22:56:37.410 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_ping_grace": "15", 2023-12-15T22:56:37.410 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_ping_interval": "5", 2023-12-15T22:56:37.410 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2023-12-15T22:56:37.410 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_recall_global_max_decay_threshold": "131072", 2023-12-15T22:56:37.410 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_recall_max_caps": "30000", 2023-12-15T22:56:37.411 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_recall_max_decay_rate": "1.500000", 2023-12-15T22:56:37.411 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_recall_max_decay_threshold": "131072", 2023-12-15T22:56:37.411 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_recall_warning_decay_rate": "60.000000", 2023-12-15T22:56:37.411 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_recall_warning_threshold": "262144", 2023-12-15T22:56:37.411 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_reconnect_timeout": "45.000000", 2023-12-15T22:56:37.411 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_replay_interval": "1.000000", 2023-12-15T22:56:37.411 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_replay_unsafe_with_closed_session": "false", 2023-12-15T22:56:37.411 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_request_load_average_decay_rate": "60.000000", 2023-12-15T22:56:37.411 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_root_ino_gid": "0", 2023-12-15T22:56:37.411 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_root_ino_uid": "0", 2023-12-15T22:56:37.411 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_scatter_nudge_interval": "5.000000", 2023-12-15T22:56:37.411 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_session_blocklist_on_evict": "true", 2023-12-15T22:56:37.411 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_session_blocklist_on_timeout": "true", 2023-12-15T22:56:37.411 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2023-12-15T22:56:37.412 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_session_cache_liveness_magnitude": "10", 2023-12-15T22:56:37.412 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2023-12-15T22:56:37.412 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_session_cap_acquisition_throttle": "100000", 2023-12-15T22:56:37.412 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2023-12-15T22:56:37.412 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_session_metadata_threshold": "16777216", 2023-12-15T22:56:37.412 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_sessionmap_keys_per_op": "1024", 2023-12-15T22:56:37.412 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_shutdown_check": "0", 2023-12-15T22:56:37.412 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_skip_ino": "0", 2023-12-15T22:56:37.412 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_sleep_rank_change": "0.000000", 2023-12-15T22:56:37.412 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_snap_max_uid": "4294967294", 2023-12-15T22:56:37.412 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_snap_min_uid": "0", 2023-12-15T22:56:37.412 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_snap_rstat": "false", 2023-12-15T22:56:37.412 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_standby_replay_damaged": "false", 2023-12-15T22:56:37.412 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_symlink_recovery": "true", 2023-12-15T22:56:37.412 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_task_status_update_interval": "2.000000", 2023-12-15T22:56:37.413 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_thrash_exports": "0", 2023-12-15T22:56:37.413 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_thrash_fragments": "0", 2023-12-15T22:56:37.413 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_tick_interval": "5.000000", 2023-12-15T22:56:37.413 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_valgrind_exit": "false", 2023-12-15T22:56:37.413 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_verify_backtrace": "1", 2023-12-15T22:56:37.413 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_verify_scatter": "false", 2023-12-15T22:56:37.413 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_wipe_ino_prealloc": "false", 2023-12-15T22:56:37.413 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_wipe_sessions": "false", 2023-12-15T22:56:37.413 INFO:teuthology.orchestra.run.smithi071.stdout: "mempool_debug": "false", 2023-12-15T22:56:37.413 INFO:teuthology.orchestra.run.smithi071.stdout: "memstore_debug_omit_block_device_write": "false", 2023-12-15T22:56:37.413 INFO:teuthology.orchestra.run.smithi071.stdout: "memstore_device_bytes": "1073741824", 2023-12-15T22:56:37.413 INFO:teuthology.orchestra.run.smithi071.stdout: "memstore_page_set": "false", 2023-12-15T22:56:37.413 INFO:teuthology.orchestra.run.smithi071.stdout: "memstore_page_size": "65536", 2023-12-15T22:56:37.414 INFO:teuthology.orchestra.run.smithi071.stdout: "mgr_client_bytes": "134217728", 2023-12-15T22:56:37.414 INFO:teuthology.orchestra.run.smithi071.stdout: "mgr_client_messages": "512", 2023-12-15T22:56:37.414 INFO:teuthology.orchestra.run.smithi071.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2023-12-15T22:56:37.414 INFO:teuthology.orchestra.run.smithi071.stdout: "mgr_connect_retry_interval": "1.000000", 2023-12-15T22:56:37.414 INFO:teuthology.orchestra.run.smithi071.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2023-12-15T22:56:37.414 INFO:teuthology.orchestra.run.smithi071.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2023-12-15T22:56:37.414 INFO:teuthology.orchestra.run.smithi071.stdout: "mgr_disabled_modules": "", 2023-12-15T22:56:37.414 INFO:teuthology.orchestra.run.smithi071.stdout: "mgr_initial_modules": "restful iostat nfs", 2023-12-15T22:56:37.414 INFO:teuthology.orchestra.run.smithi071.stdout: "mgr_max_pg_num_change": "128", 2023-12-15T22:56:37.414 INFO:teuthology.orchestra.run.smithi071.stdout: "mgr_mds_bytes": "134217728", 2023-12-15T22:56:37.414 INFO:teuthology.orchestra.run.smithi071.stdout: "mgr_mds_messages": "128", 2023-12-15T22:56:37.414 INFO:teuthology.orchestra.run.smithi071.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2023-12-15T22:56:37.414 INFO:teuthology.orchestra.run.smithi071.stdout: "mgr_mon_bytes": "134217728", 2023-12-15T22:56:37.414 INFO:teuthology.orchestra.run.smithi071.stdout: "mgr_mon_messages": "128", 2023-12-15T22:56:37.415 INFO:teuthology.orchestra.run.smithi071.stdout: "mgr_osd_bytes": "536870912", 2023-12-15T22:56:37.415 INFO:teuthology.orchestra.run.smithi071.stdout: "mgr_osd_messages": "8192", 2023-12-15T22:56:37.415 INFO:teuthology.orchestra.run.smithi071.stdout: "mgr_pool": "true", 2023-12-15T22:56:37.415 INFO:teuthology.orchestra.run.smithi071.stdout: "mgr_service_beacon_grace": "60.000000", 2023-12-15T22:56:37.415 INFO:teuthology.orchestra.run.smithi071.stdout: "mgr_standby_modules": "true", 2023-12-15T22:56:37.415 INFO:teuthology.orchestra.run.smithi071.stdout: "mgr_stats_period": "5", 2023-12-15T22:56:37.415 INFO:teuthology.orchestra.run.smithi071.stdout: "mgr_stats_threshold": "5", 2023-12-15T22:56:37.415 INFO:teuthology.orchestra.run.smithi071.stdout: "mgr_tick_period": "2", 2023-12-15T22:56:37.415 INFO:teuthology.orchestra.run.smithi071.stdout: "mgr_ttl_cache_expire_seconds": "0", 2023-12-15T22:56:37.415 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_accept_timeout_factor": "2.000000", 2023-12-15T22:56:37.415 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_allow_pool_delete": "true", 2023-12-15T22:56:37.415 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_allow_pool_size_one": "true", 2023-12-15T22:56:37.415 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_auth_validate_all_caps": "true", 2023-12-15T22:56:37.416 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2023-12-15T22:56:37.416 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2023-12-15T22:56:37.416 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_client_bytes": "104857600", 2023-12-15T22:56:37.416 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_client_directed_command_retry": "2", 2023-12-15T22:56:37.416 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_client_hunt_interval": "3.000000", 2023-12-15T22:56:37.416 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2023-12-15T22:56:37.416 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2023-12-15T22:56:37.416 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2023-12-15T22:56:37.416 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_client_hunt_parallel": "3", 2023-12-15T22:56:37.416 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_client_log_interval": "1.000000", 2023-12-15T22:56:37.416 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_client_max_log_entries_per_message": "1000", 2023-12-15T22:56:37.416 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_client_ping_interval": "10.000000", 2023-12-15T22:56:37.416 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_client_ping_timeout": "30.000000", 2023-12-15T22:56:37.416 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_client_target_rank": "-1", 2023-12-15T22:56:37.416 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_clock_drift_allowed": "1.000000", 2023-12-15T22:56:37.417 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2023-12-15T22:56:37.417 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2023-12-15T22:56:37.417 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_cluster_log_file_level": "debug", 2023-12-15T22:56:37.417 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_cluster_log_to_file": "true", 2023-12-15T22:56:37.417 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_cluster_log_to_graylog": "false", 2023-12-15T22:56:37.417 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2023-12-15T22:56:37.417 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_cluster_log_to_graylog_port": "12201", 2023-12-15T22:56:37.417 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_cluster_log_to_journald": "false", 2023-12-15T22:56:37.417 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_cluster_log_to_stderr": "false", 2023-12-15T22:56:37.417 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_cluster_log_to_syslog": "default=false", 2023-12-15T22:56:37.417 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2023-12-15T22:56:37.417 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_cluster_log_to_syslog_level": "info", 2023-12-15T22:56:37.417 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_compact_on_bootstrap": "false", 2023-12-15T22:56:37.417 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_compact_on_start": "false", 2023-12-15T22:56:37.417 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_compact_on_trim": "true", 2023-12-15T22:56:37.418 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_con_tracker_persist_interval": "10", 2023-12-15T22:56:37.418 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_con_tracker_score_halflife": "43200", 2023-12-15T22:56:37.418 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_config_key_max_entry_size": "65536", 2023-12-15T22:56:37.418 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_cpu_threads": "4", 2023-12-15T22:56:37.418 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_crush_min_required_version": "hammer", 2023-12-15T22:56:37.418 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_daemon_bytes": "419430400", 2023-12-15T22:56:37.418 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2023-12-15T22:56:37.418 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_data_avail_crit": "5", 2023-12-15T22:56:37.418 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_data_avail_warn": "5", 2023-12-15T22:56:37.418 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_data_size_warn": "16106127360", 2023-12-15T22:56:37.418 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_debug_block_osdmap_trim": "false", 2023-12-15T22:56:37.418 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_debug_deprecated_as_obsolete": "false", 2023-12-15T22:56:37.418 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_debug_dump_json": "false", 2023-12-15T22:56:37.418 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2023-12-15T22:56:37.418 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_debug_dump_transactions": "false", 2023-12-15T22:56:37.419 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_debug_extra_checks": "false", 2023-12-15T22:56:37.419 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_debug_no_initial_persistent_features": "false", 2023-12-15T22:56:37.419 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2023-12-15T22:56:37.419 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_debug_no_require_quincy": "false", 2023-12-15T22:56:37.419 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_debug_no_require_reef": "false", 2023-12-15T22:56:37.419 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2023-12-15T22:56:37.419 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_delta_reset_interval": "10.000000", 2023-12-15T22:56:37.419 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_dns_srv_name": "ceph-mon", 2023-12-15T22:56:37.419 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_down_mkfs_grace": "120", 2023-12-15T22:56:37.419 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_election_default_strategy": "1", 2023-12-15T22:56:37.419 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_election_timeout": "5.000000", 2023-12-15T22:56:37.419 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2023-12-15T22:56:37.419 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_elector_ping_divisor": "2", 2023-12-15T22:56:37.420 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_elector_ping_timeout": "2.000000", 2023-12-15T22:56:37.420 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_enable_op_tracker": "true", 2023-12-15T22:56:37.420 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_fake_pool_delete": "false", 2023-12-15T22:56:37.420 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_force_quorum_join": "false", 2023-12-15T22:56:37.420 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_globalid_prealloc": "10000", 2023-12-15T22:56:37.420 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_health_detail_to_clog": "false", 2023-12-15T22:56:37.420 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_health_log_update_period": "5", 2023-12-15T22:56:37.420 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_health_max_detail": "50", 2023-12-15T22:56:37.420 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_health_to_clog": "true", 2023-12-15T22:56:37.420 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_health_to_clog_interval": "600", 2023-12-15T22:56:37.420 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2023-12-15T22:56:37.420 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_host": "172.21.15.71", 2023-12-15T22:56:37.420 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_host_override": "", 2023-12-15T22:56:37.420 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_initial_members": "", 2023-12-15T22:56:37.421 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2023-12-15T22:56:37.421 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2023-12-15T22:56:37.421 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_inject_transaction_delay_max": "10.000000", 2023-12-15T22:56:37.421 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2023-12-15T22:56:37.421 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_keyvaluedb": "rocksdb", 2023-12-15T22:56:37.421 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_lease": "5.000000", 2023-12-15T22:56:37.421 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2023-12-15T22:56:37.421 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_lease_renew_interval_factor": "0.600000", 2023-12-15T22:56:37.421 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_log_full_interval": "50", 2023-12-15T22:56:37.421 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_log_max": "10000", 2023-12-15T22:56:37.421 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_log_max_summary": "50", 2023-12-15T22:56:37.421 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_max_log_entries_per_event": "4096", 2023-12-15T22:56:37.421 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_max_log_epochs": "500", 2023-12-15T22:56:37.421 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_max_mdsmap_epochs": "500", 2023-12-15T22:56:37.422 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_max_mgrmap_epochs": "500", 2023-12-15T22:56:37.422 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_max_osd": "10000", 2023-12-15T22:56:37.422 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_max_pg_per_osd": "10000", 2023-12-15T22:56:37.422 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_max_pool_pg_num": "65536", 2023-12-15T22:56:37.422 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_max_snap_prune_per_epoch": "100", 2023-12-15T22:56:37.422 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_mds_blocklist_interval": "86400.000000", 2023-12-15T22:56:37.422 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_mds_force_trim_to": "0", 2023-12-15T22:56:37.422 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_mds_skip_sanity": "false", 2023-12-15T22:56:37.422 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_memory_autotune": "true", 2023-12-15T22:56:37.422 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_memory_target": "2147483648", 2023-12-15T22:56:37.422 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_mgr_beacon_grace": "30", 2023-12-15T22:56:37.422 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2023-12-15T22:56:37.422 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_mgr_digest_period": "5", 2023-12-15T22:56:37.422 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_mgr_inactive_grace": "60", 2023-12-15T22:56:37.423 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_mgr_mkfs_grace": "240", 2023-12-15T22:56:37.423 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2023-12-15T22:56:37.423 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_min_osdmap_epochs": "50", 2023-12-15T22:56:37.423 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_op_complaint_time": "30", 2023-12-15T22:56:37.423 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_op_history_duration": "600", 2023-12-15T22:56:37.423 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_op_history_size": "20", 2023-12-15T22:56:37.423 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_op_history_slow_op_size": "20", 2023-12-15T22:56:37.423 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_op_history_slow_op_threshold": "10", 2023-12-15T22:56:37.423 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_op_log_threshold": "5", 2023-12-15T22:56:37.423 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_osd_adjust_down_out_interval": "true", 2023-12-15T22:56:37.423 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2023-12-15T22:56:37.423 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2023-12-15T22:56:37.423 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_osd_auto_mark_in": "false", 2023-12-15T22:56:37.423 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_osd_auto_mark_new_in": "true", 2023-12-15T22:56:37.423 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2023-12-15T22:56:37.424 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2023-12-15T22:56:37.424 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_osd_cache_size": "500", 2023-12-15T22:56:37.424 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_osd_cache_size_min": "134217728", 2023-12-15T22:56:37.424 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_osd_crush_smoke_test": "true", 2023-12-15T22:56:37.424 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_osd_destroyed_out_interval": "600", 2023-12-15T22:56:37.424 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_osd_down_out_interval": "600", 2023-12-15T22:56:37.424 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_osd_down_out_subtree_limit": "rack", 2023-12-15T22:56:37.424 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2023-12-15T22:56:37.424 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_osd_force_trim_to": "0", 2023-12-15T22:56:37.424 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_osd_full_ratio": "0.950000", 2023-12-15T22:56:37.424 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2023-12-15T22:56:37.424 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_osd_laggy_halflife": "3600", 2023-12-15T22:56:37.424 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_osd_laggy_max_interval": "300", 2023-12-15T22:56:37.424 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_osd_laggy_weight": "0.300000", 2023-12-15T22:56:37.425 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2023-12-15T22:56:37.425 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_osd_max_creating_pgs": "1024", 2023-12-15T22:56:37.425 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_osd_max_initial_pgs": "1024", 2023-12-15T22:56:37.425 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_osd_min_down_reporters": "2", 2023-12-15T22:56:37.425 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_osd_min_in_ratio": "0.750000", 2023-12-15T22:56:37.425 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_osd_min_up_ratio": "0.300000", 2023-12-15T22:56:37.425 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_osd_nearfull_ratio": "0.850000", 2023-12-15T22:56:37.425 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_osd_prime_pg_temp": "true", 2023-12-15T22:56:37.425 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2023-12-15T22:56:37.425 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2023-12-15T22:56:37.425 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_osd_report_timeout": "900", 2023-12-15T22:56:37.425 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_osd_reporter_subtree_level": "osd", 2023-12-15T22:56:37.425 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2023-12-15T22:56:37.425 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_osd_warn_num_repaired": "10", 2023-12-15T22:56:37.425 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_osd_warn_op_age": "32.000000", 2023-12-15T22:56:37.426 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_osdmap_full_prune_enabled": "true", 2023-12-15T22:56:37.426 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_osdmap_full_prune_interval": "2", 2023-12-15T22:56:37.426 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_osdmap_full_prune_min": "15", 2023-12-15T22:56:37.426 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_osdmap_full_prune_txsize": "2", 2023-12-15T22:56:37.426 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2023-12-15T22:56:37.426 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_pg_stuck_threshold": "60", 2023-12-15T22:56:37.426 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2023-12-15T22:56:37.426 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_pg_warn_min_objects": "10000", 2023-12-15T22:56:37.426 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_pg_warn_min_per_osd": "0", 2023-12-15T22:56:37.426 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_pg_warn_min_pool_objects": "1000", 2023-12-15T22:56:37.426 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_pool_quota_crit_threshold": "0", 2023-12-15T22:56:37.426 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_pool_quota_warn_threshold": "0", 2023-12-15T22:56:37.426 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_probe_timeout": "2.000000", 2023-12-15T22:56:37.426 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_reweight_max_change": "0.050000", 2023-12-15T22:56:37.427 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_reweight_max_osds": "4", 2023-12-15T22:56:37.427 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_reweight_min_bytes_per_osd": "10", 2023-12-15T22:56:37.427 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_reweight_min_pgs_per_osd": "4", 2023-12-15T22:56:37.427 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2023-12-15T22:56:37.427 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2023-12-15T22:56:37.427 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2023-12-15T22:56:37.427 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_scrub_interval": "86400", 2023-12-15T22:56:37.427 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_scrub_max_keys": "100", 2023-12-15T22:56:37.427 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_scrub_timeout": "300", 2023-12-15T22:56:37.427 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_session_timeout": "300", 2023-12-15T22:56:37.427 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_smart_report_timeout": "5", 2023-12-15T22:56:37.427 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_stat_smooth_intervals": "6", 2023-12-15T22:56:37.427 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2023-12-15T22:56:37.427 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_stretch_pool_min_size": "2", 2023-12-15T22:56:37.427 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_stretch_pool_size": "4", 2023-12-15T22:56:37.428 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2023-12-15T22:56:37.428 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_subscribe_interval": "86400.000000", 2023-12-15T22:56:37.428 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_sync_debug": "false", 2023-12-15T22:56:37.428 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_sync_max_payload_keys": "2000", 2023-12-15T22:56:37.428 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_sync_max_payload_size": "1048576", 2023-12-15T22:56:37.428 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_sync_provider_kill_at": "0", 2023-12-15T22:56:37.428 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_sync_requester_kill_at": "0", 2023-12-15T22:56:37.428 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_sync_timeout": "60.000000", 2023-12-15T22:56:37.428 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_target_pg_per_osd": "100", 2023-12-15T22:56:37.428 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_tick_interval": "5", 2023-12-15T22:56:37.428 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_timecheck_interval": "300.000000", 2023-12-15T22:56:37.428 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_timecheck_skew_interval": "30.000000", 2023-12-15T22:56:37.428 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_use_min_delay_socket": "false", 2023-12-15T22:56:37.428 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_warn_older_version_delay": "604800", 2023-12-15T22:56:37.429 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2023-12-15T22:56:37.429 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2023-12-15T22:56:37.429 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2023-12-15T22:56:37.429 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_warn_on_filestore_osds": "false", 2023-12-15T22:56:37.429 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2023-12-15T22:56:37.429 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2023-12-15T22:56:37.429 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2023-12-15T22:56:37.429 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_warn_on_misplaced": "false", 2023-12-15T22:56:37.429 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2023-12-15T22:56:37.429 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_warn_on_older_version": "true", 2023-12-15T22:56:37.429 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2023-12-15T22:56:37.430 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_warn_on_pool_no_app": "true", 2023-12-15T22:56:37.430 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_warn_on_pool_no_redundancy": "false", 2023-12-15T22:56:37.430 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2023-12-15T22:56:37.430 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2023-12-15T22:56:37.430 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2023-12-15T22:56:37.430 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_warn_on_too_few_osds": "false", 2023-12-15T22:56:37.430 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2023-12-15T22:56:37.430 INFO:teuthology.orchestra.run.smithi071.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2023-12-15T22:56:37.430 INFO:teuthology.orchestra.run.smithi071.stdout: "monmap": "", 2023-12-15T22:56:37.430 INFO:teuthology.orchestra.run.smithi071.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2023-12-15T22:56:37.430 INFO:teuthology.orchestra.run.smithi071.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2023-12-15T22:56:37.430 INFO:teuthology.orchestra.run.smithi071.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2023-12-15T22:56:37.431 INFO:teuthology.orchestra.run.smithi071.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2023-12-15T22:56:37.431 INFO:teuthology.orchestra.run.smithi071.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2023-12-15T22:56:37.431 INFO:teuthology.orchestra.run.smithi071.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2023-12-15T22:56:37.431 INFO:teuthology.orchestra.run.smithi071.stdout: "motr_tracing_enabled": "false", 2023-12-15T22:56:37.431 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_async_op_threads": "3", 2023-12-15T22:56:37.431 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_async_rdma_buffer_size": "131072", 2023-12-15T22:56:37.431 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_async_rdma_cm": "false", 2023-12-15T22:56:37.431 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_async_rdma_device_name": "", 2023-12-15T22:56:37.431 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_async_rdma_dscp": "96", 2023-12-15T22:56:37.431 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_async_rdma_enable_hugepage": "false", 2023-12-15T22:56:37.432 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_async_rdma_gid_idx": "0", 2023-12-15T22:56:37.432 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_async_rdma_local_gid": "", 2023-12-15T22:56:37.432 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_async_rdma_polling_us": "1000", 2023-12-15T22:56:37.432 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_async_rdma_port_num": "1", 2023-12-15T22:56:37.432 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_async_rdma_receive_buffers": "32768", 2023-12-15T22:56:37.432 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_async_rdma_receive_queue_len": "4096", 2023-12-15T22:56:37.432 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_async_rdma_roce_ver": "1", 2023-12-15T22:56:37.432 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_async_rdma_send_buffers": "1024", 2023-12-15T22:56:37.432 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_async_rdma_sl": "3", 2023-12-15T22:56:37.432 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_async_rdma_support_srq": "true", 2023-12-15T22:56:37.432 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_async_rdma_type": "ib", 2023-12-15T22:56:37.432 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_async_reap_threshold": "5", 2023-12-15T22:56:37.432 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_bind_before_connect": "false", 2023-12-15T22:56:37.432 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_bind_ipv4": "true", 2023-12-15T22:56:37.433 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_bind_ipv6": "false", 2023-12-15T22:56:37.433 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_bind_msgr1": "true", 2023-12-15T22:56:37.433 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_bind_msgr2": "true", 2023-12-15T22:56:37.433 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_bind_port_max": "7568", 2023-12-15T22:56:37.433 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_bind_port_min": "6800", 2023-12-15T22:56:37.433 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_bind_prefer_ipv4": "false", 2023-12-15T22:56:37.433 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_bind_retry_count": "3", 2023-12-15T22:56:37.433 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_bind_retry_delay": "5", 2023-12-15T22:56:37.433 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_blackhole_client": "false", 2023-12-15T22:56:37.433 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_blackhole_mds": "false", 2023-12-15T22:56:37.434 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_blackhole_mgr": "false", 2023-12-15T22:56:37.434 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_blackhole_mon": "false", 2023-12-15T22:56:37.434 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_blackhole_osd": "false", 2023-12-15T22:56:37.434 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_client_mode": "crc", 2023-12-15T22:56:37.434 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_cluster_mode": "crc", 2023-12-15T22:56:37.434 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_cluster_type": "", 2023-12-15T22:56:37.434 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_compress_secure": "false", 2023-12-15T22:56:37.434 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_connection_idle_timeout": "900", 2023-12-15T22:56:37.434 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_connection_ready_timeout": "10", 2023-12-15T22:56:37.434 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_crc_data": "true", 2023-12-15T22:56:37.434 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_crc_header": "true", 2023-12-15T22:56:37.434 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_die_on_bad_msg": "false", 2023-12-15T22:56:37.434 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_die_on_bug": "true", 2023-12-15T22:56:37.435 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_die_on_old_message": "true", 2023-12-15T22:56:37.435 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_die_on_skipped_message": "false", 2023-12-15T22:56:37.435 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_die_on_unhandled_msg": "false", 2023-12-15T22:56:37.435 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_dispatch_throttle_bytes": "104857600", 2023-12-15T22:56:37.435 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_dpdk_coremask": "0xF", 2023-12-15T22:56:37.435 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_dpdk_debug_allow_loopback": "false", 2023-12-15T22:56:37.435 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_dpdk_devs_allowlist": "", 2023-12-15T22:56:37.435 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_dpdk_enable_tso": "true", 2023-12-15T22:56:37.435 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2023-12-15T22:56:37.435 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_dpdk_host_ipv4_addr": "", 2023-12-15T22:56:37.436 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_dpdk_hugepages": "", 2023-12-15T22:56:37.436 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_dpdk_hw_flow_control": "true", 2023-12-15T22:56:37.436 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2023-12-15T22:56:37.436 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_dpdk_lro": "true", 2023-12-15T22:56:37.436 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_dpdk_memory_channel": "4", 2023-12-15T22:56:37.436 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2023-12-15T22:56:37.436 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_dpdk_pmd": "", 2023-12-15T22:56:37.436 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_dpdk_port_id": "0", 2023-12-15T22:56:37.436 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2023-12-15T22:56:37.436 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_dump_corrupt_message_level": "1", 2023-12-15T22:56:37.437 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_dump_on_send": "false", 2023-12-15T22:56:37.437 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_initial_backoff": "0.200000", 2023-12-15T22:56:37.437 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_inject_delay_max": "1.000000", 2023-12-15T22:56:37.437 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_inject_delay_probability": "0.000000", 2023-12-15T22:56:37.437 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_inject_delay_type": "", 2023-12-15T22:56:37.437 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_inject_internal_delays": "0.000000", 2023-12-15T22:56:37.437 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_inject_network_congestion": "0", 2023-12-15T22:56:37.437 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_inject_socket_failures": "0", 2023-12-15T22:56:37.437 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_learn_addr_from_peer": "true", 2023-12-15T22:56:37.437 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_max_accept_failures": "4", 2023-12-15T22:56:37.437 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_max_backoff": "15.000000", 2023-12-15T22:56:37.437 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_mon_client_mode": "crc", 2023-12-15T22:56:37.437 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_mon_cluster_mode": "crc", 2023-12-15T22:56:37.437 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_mon_service_mode": "crc", 2023-12-15T22:56:37.438 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_osd_compress_min_size": "1024", 2023-12-15T22:56:37.438 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_osd_compress_mode": "none", 2023-12-15T22:56:37.438 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_osd_compression_algorithm": "snappy", 2023-12-15T22:56:37.438 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2023-12-15T22:56:37.438 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_pq_min_cost": "65536", 2023-12-15T22:56:37.438 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_public_type": "", 2023-12-15T22:56:37.438 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_service_mode": "crc", 2023-12-15T22:56:37.438 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_tcp_listen_backlog": "512", 2023-12-15T22:56:37.438 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_tcp_nodelay": "true", 2023-12-15T22:56:37.438 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_tcp_prefetch_max_size": "4096", 2023-12-15T22:56:37.438 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_tcp_rcvbuf": "0", 2023-12-15T22:56:37.438 INFO:teuthology.orchestra.run.smithi071.stdout: "ms_type": "async+posix", 2023-12-15T22:56:37.438 INFO:teuthology.orchestra.run.smithi071.stdout: "no_config_file": "false", 2023-12-15T22:56:37.438 INFO:teuthology.orchestra.run.smithi071.stdout: "objecter_completion_locks_per_session": "32", 2023-12-15T22:56:37.438 INFO:teuthology.orchestra.run.smithi071.stdout: "objecter_debug_inject_relock_delay": "false", 2023-12-15T22:56:37.439 INFO:teuthology.orchestra.run.smithi071.stdout: "objecter_inflight_op_bytes": "104857600", 2023-12-15T22:56:37.439 INFO:teuthology.orchestra.run.smithi071.stdout: "objecter_inflight_ops": "1024", 2023-12-15T22:56:37.439 INFO:teuthology.orchestra.run.smithi071.stdout: "objecter_inject_no_watch_ping": "false", 2023-12-15T22:56:37.439 INFO:teuthology.orchestra.run.smithi071.stdout: "objecter_retry_writes_after_first_reply": "false", 2023-12-15T22:56:37.439 INFO:teuthology.orchestra.run.smithi071.stdout: "objecter_tick_interval": "5.000000", 2023-12-15T22:56:37.439 INFO:teuthology.orchestra.run.smithi071.stdout: "objecter_timeout": "10.000000", 2023-12-15T22:56:37.439 INFO:teuthology.orchestra.run.smithi071.stdout: "objectstore_blackhole": "false", 2023-12-15T22:56:37.439 INFO:teuthology.orchestra.run.smithi071.stdout: "openssl_engine_opts": "", 2023-12-15T22:56:37.439 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_agent_delay_time": "5.000000", 2023-12-15T22:56:37.439 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_agent_hist_halflife": "1000", 2023-12-15T22:56:37.439 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_agent_max_low_ops": "2", 2023-12-15T22:56:37.439 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_agent_max_ops": "4", 2023-12-15T22:56:37.439 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_agent_min_evict_effort": "0.100000", 2023-12-15T22:56:37.439 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_agent_quantize_effort": "0.100000", 2023-12-15T22:56:37.439 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_agent_slop": "0.020000", 2023-12-15T22:56:37.440 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_aggregated_slow_ops_logging": "true", 2023-12-15T22:56:37.440 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_allow_recovery_below_min_size": "true", 2023-12-15T22:56:37.440 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_asio_thread_count": "2", 2023-12-15T22:56:37.440 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_async_recovery_min_cost": "100", 2023-12-15T22:56:37.440 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_auto_mark_unfound_lost": "false", 2023-12-15T22:56:37.440 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_backfill_retry_interval": "30.000000", 2023-12-15T22:56:37.440 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_backfill_scan_max": "512", 2023-12-15T22:56:37.440 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_backfill_scan_min": "64", 2023-12-15T22:56:37.440 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_backoff_on_degraded": "false", 2023-12-15T22:56:37.440 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_backoff_on_peering": "false", 2023-12-15T22:56:37.440 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_backoff_on_unfound": "true", 2023-12-15T22:56:37.440 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_beacon_report_interval": "300", 2023-12-15T22:56:37.440 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_bench_duration": "30", 2023-12-15T22:56:37.440 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_bench_large_size_max_throughput": "104857600", 2023-12-15T22:56:37.440 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_bench_max_block_size": "67108864", 2023-12-15T22:56:37.441 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_bench_small_size_max_iops": "100", 2023-12-15T22:56:37.441 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_blkin_trace_all": "false", 2023-12-15T22:56:37.441 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_blocked_scrub_grace_period": "120", 2023-12-15T22:56:37.441 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2023-12-15T22:56:37.441 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2023-12-15T22:56:37.441 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2023-12-15T22:56:37.441 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_check_for_log_corruption": "false", 2023-12-15T22:56:37.441 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_check_max_object_name_len_on_startup": "true", 2023-12-15T22:56:37.441 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2023-12-15T22:56:37.441 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2023-12-15T22:56:37.441 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2023-12-15T22:56:37.441 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_class_update_on_start": "true", 2023-12-15T22:56:37.441 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_client_message_cap": "256", 2023-12-15T22:56:37.441 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_client_message_size_cap": "524288000", 2023-12-15T22:56:37.442 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_client_op_priority": "63", 2023-12-15T22:56:37.442 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_client_watch_timeout": "30", 2023-12-15T22:56:37.442 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_command_max_records": "256", 2023-12-15T22:56:37.442 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_compact_on_start": "false", 2023-12-15T22:56:37.442 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_copyfrom_max_chunk": "8388608", 2023-12-15T22:56:37.442 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_crush_chooseleaf_type": "0", 2023-12-15T22:56:37.442 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_crush_initial_weight": "-1.000000", 2023-12-15T22:56:37.442 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_crush_update_on_start": "true", 2023-12-15T22:56:37.442 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_crush_update_weight_set": "true", 2023-12-15T22:56:37.442 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2023-12-15T22:56:37.442 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2023-12-15T22:56:37.442 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2023-12-15T22:56:37.442 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_debug_drop_ping_duration": "0", 2023-12-15T22:56:37.442 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_debug_drop_ping_probability": "0.000000", 2023-12-15T22:56:37.442 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_debug_feed_pullee": "-1", 2023-12-15T22:56:37.443 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_debug_inject_copyfrom_error": "false", 2023-12-15T22:56:37.443 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2023-12-15T22:56:37.443 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2023-12-15T22:56:37.443 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_debug_misdirected_ops": "false", 2023-12-15T22:56:37.443 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_debug_no_acting_change": "false", 2023-12-15T22:56:37.443 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_debug_no_purge_strays": "false", 2023-12-15T22:56:37.443 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_debug_op_order": "false", 2023-12-15T22:56:37.443 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_debug_pg_log_writeout": "false", 2023-12-15T22:56:37.443 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_debug_pretend_recovery_active": "false", 2023-12-15T22:56:37.443 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_debug_random_push_read_error": "0.000000", 2023-12-15T22:56:37.443 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2023-12-15T22:56:37.443 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_debug_shutdown": "false", 2023-12-15T22:56:37.443 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2023-12-15T22:56:37.443 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2023-12-15T22:56:37.443 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_debug_trim_objects": "false", 2023-12-15T22:56:37.444 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_debug_verify_cached_snaps": "false", 2023-12-15T22:56:37.444 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_debug_verify_missing_on_start": "false", 2023-12-15T22:56:37.444 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_debug_verify_snaps": "false", 2023-12-15T22:56:37.444 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_debug_verify_stray_on_activate": "false", 2023-12-15T22:56:37.444 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_deep_scrub_interval": "604800.000000", 2023-12-15T22:56:37.444 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_deep_scrub_keys": "1024", 2023-12-15T22:56:37.444 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2023-12-15T22:56:37.444 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2023-12-15T22:56:37.444 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2023-12-15T22:56:37.444 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_deep_scrub_stride": "524288", 2023-12-15T22:56:37.444 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2023-12-15T22:56:37.444 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_default_data_pool_replay_window": "5", 2023-12-15T22:56:37.444 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_default_notify_timeout": "30", 2023-12-15T22:56:37.444 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_delete_sleep": "0.000000", 2023-12-15T22:56:37.445 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_delete_sleep_hdd": "5.000000", 2023-12-15T22:56:37.445 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_delete_sleep_hybrid": "1.000000", 2023-12-15T22:56:37.445 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_delete_sleep_ssd": "1.000000", 2023-12-15T22:56:37.445 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_discard_disconnected_ops": "true", 2023-12-15T22:56:37.445 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_enable_op_tracker": "true", 2023-12-15T22:56:37.445 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2023-12-15T22:56:37.445 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_extblkdev_plugins": "vdo", 2023-12-15T22:56:37.445 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_failsafe_full_ratio": "0.970000", 2023-12-15T22:56:37.445 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_fast_fail_on_connection_refused": "true", 2023-12-15T22:56:37.445 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_fast_info": "true", 2023-12-15T22:56:37.445 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_fast_shutdown": "true", 2023-12-15T22:56:37.445 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_fast_shutdown_notify_mon": "true", 2023-12-15T22:56:37.445 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_fast_shutdown_timeout": "15", 2023-12-15T22:56:37.445 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_find_best_info_ignore_history_les": "false", 2023-12-15T22:56:37.445 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_force_auth_primary_missing_objects": "100", 2023-12-15T22:56:37.446 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2023-12-15T22:56:37.446 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_function_tracing": "false", 2023-12-15T22:56:37.446 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_heartbeat_grace": "20", 2023-12-15T22:56:37.446 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_heartbeat_interval": "6", 2023-12-15T22:56:37.446 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2023-12-15T22:56:37.446 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_heartbeat_min_peers": "10", 2023-12-15T22:56:37.446 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_heartbeat_min_size": "2000", 2023-12-15T22:56:37.446 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_heartbeat_stale": "600", 2023-12-15T22:56:37.446 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2023-12-15T22:56:37.446 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_hit_set_max_size": "100000", 2023-12-15T22:56:37.446 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_hit_set_min_size": "1000", 2023-12-15T22:56:37.446 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_hit_set_namespace": ".ceph-internal", 2023-12-15T22:56:37.446 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_ignore_stale_divergent_priors": "false", 2023-12-15T22:56:37.446 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2023-12-15T22:56:37.447 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_inject_failure_on_pg_removal": "false", 2023-12-15T22:56:37.447 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2023-12-15T22:56:37.447 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_journal_flush_on_shutdown": "true", 2023-12-15T22:56:37.447 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_journal_size": "5120", 2023-12-15T22:56:37.447 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_kill_backfill_at": "0", 2023-12-15T22:56:37.447 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_loop_before_reset_tphandle": "64", 2023-12-15T22:56:37.447 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_map_cache_size": "50", 2023-12-15T22:56:37.447 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_map_dedup": "true", 2023-12-15T22:56:37.447 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_map_message_max": "40", 2023-12-15T22:56:37.447 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_map_message_max_bytes": "10485760", 2023-12-15T22:56:37.447 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_map_share_max_epochs": "40", 2023-12-15T22:56:37.447 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_max_attr_name_len": "100", 2023-12-15T22:56:37.447 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_max_attr_size": "0", 2023-12-15T22:56:37.447 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_max_backfills": "1", 2023-12-15T22:56:37.447 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_max_markdown_count": "5", 2023-12-15T22:56:37.448 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_max_markdown_period": "600", 2023-12-15T22:56:37.448 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_max_object_name_len": "2048", 2023-12-15T22:56:37.448 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_max_object_namespace_len": "256", 2023-12-15T22:56:37.448 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_max_object_size": "134217728", 2023-12-15T22:56:37.448 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2023-12-15T22:56:37.448 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_max_omap_entries_per_request": "1024", 2023-12-15T22:56:37.448 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_max_pg_blocked_by": "16", 2023-12-15T22:56:37.448 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_max_pg_log_entries": "10000", 2023-12-15T22:56:37.448 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2023-12-15T22:56:37.448 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_max_pgls": "1024", 2023-12-15T22:56:37.448 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_max_push_cost": "8388608", 2023-12-15T22:56:37.448 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_max_push_objects": "10", 2023-12-15T22:56:37.448 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_max_scrubs": "3", 2023-12-15T22:56:37.448 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2023-12-15T22:56:37.448 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_max_trimming_pgs": "2", 2023-12-15T22:56:37.449 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_max_write_op_reply_len": "64", 2023-12-15T22:56:37.449 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_max_write_size": "90", 2023-12-15T22:56:37.449 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2023-12-15T22:56:37.449 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2023-12-15T22:56:37.449 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2023-12-15T22:56:37.449 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2023-12-15T22:56:37.449 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2023-12-15T22:56:37.449 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2023-12-15T22:56:37.449 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2023-12-15T22:56:37.449 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_mclock_override_recovery_settings": "false", 2023-12-15T22:56:37.449 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_mclock_profile": "balanced", 2023-12-15T22:56:37.449 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2023-12-15T22:56:37.449 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2023-12-15T22:56:37.449 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2023-12-15T22:56:37.450 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2023-12-15T22:56:37.450 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2023-12-15T22:56:37.450 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2023-12-15T22:56:37.450 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2023-12-15T22:56:37.450 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2023-12-15T22:56:37.450 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2023-12-15T22:56:37.450 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_mclock_scheduler_client_wgt": "1", 2023-12-15T22:56:37.450 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_mclock_skip_benchmark": "false", 2023-12-15T22:56:37.450 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_memory_base": "805306368", 2023-12-15T22:56:37.450 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_memory_cache_min": "134217728", 2023-12-15T22:56:37.450 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_memory_cache_resize_interval": "1.000000", 2023-12-15T22:56:37.450 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_memory_expected_fragmentation": "0.150000", 2023-12-15T22:56:37.450 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_memory_target": "4294967296", 2023-12-15T22:56:37.450 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_memory_target_autotune": "false", 2023-12-15T22:56:37.450 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2023-12-15T22:56:37.451 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_min_pg_log_entries": "250", 2023-12-15T22:56:37.451 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_min_recovery_priority": "0", 2023-12-15T22:56:37.451 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_mon_heartbeat_interval": "30", 2023-12-15T22:56:37.451 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2023-12-15T22:56:37.451 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_mon_report_interval": "5", 2023-12-15T22:56:37.451 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_mon_report_max_in_flight": "2", 2023-12-15T22:56:37.451 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_mon_shutdown_timeout": "5.000000", 2023-12-15T22:56:37.451 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_num_cache_shards": "32", 2023-12-15T22:56:37.451 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_num_op_tracker_shard": "32", 2023-12-15T22:56:37.451 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_numa_auto_affinity": "true", 2023-12-15T22:56:37.451 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_numa_node": "-1", 2023-12-15T22:56:37.451 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_numa_prefer_iface": "true", 2023-12-15T22:56:37.451 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_object_clean_region_max_num_intervals": "10", 2023-12-15T22:56:37.451 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_objecter_finishers": "1", 2023-12-15T22:56:37.452 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_objectstore": "bluestore", 2023-12-15T22:56:37.452 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_objectstore_fuse": "false", 2023-12-15T22:56:37.452 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_objectstore_tracing": "false", 2023-12-15T22:56:37.452 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_op_complaint_time": "30.000000", 2023-12-15T22:56:37.452 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_op_history_duration": "600", 2023-12-15T22:56:37.452 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_op_history_size": "20", 2023-12-15T22:56:37.452 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_op_history_slow_op_size": "20", 2023-12-15T22:56:37.452 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2023-12-15T22:56:37.452 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_op_log_threshold": "5", 2023-12-15T22:56:37.452 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_op_num_shards": "0", 2023-12-15T22:56:37.452 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_op_num_shards_hdd": "5", 2023-12-15T22:56:37.452 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_op_num_shards_ssd": "8", 2023-12-15T22:56:37.452 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_op_num_threads_per_shard": "0", 2023-12-15T22:56:37.452 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2023-12-15T22:56:37.452 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2023-12-15T22:56:37.453 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2023-12-15T22:56:37.453 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_op_pq_min_cost": "65536", 2023-12-15T22:56:37.453 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_op_queue": "mclock_scheduler", 2023-12-15T22:56:37.453 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_op_queue_cut_off": "high", 2023-12-15T22:56:37.453 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_op_thread_suicide_timeout": "150", 2023-12-15T22:56:37.453 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_op_thread_timeout": "15", 2023-12-15T22:56:37.453 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_open_classes_on_start": "true", 2023-12-15T22:56:37.453 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_os_flags": "0", 2023-12-15T22:56:37.453 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_peering_op_priority": "255", 2023-12-15T22:56:37.453 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pg_delete_cost": "1048576", 2023-12-15T22:56:37.453 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pg_delete_priority": "5", 2023-12-15T22:56:37.453 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2023-12-15T22:56:37.453 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2023-12-15T22:56:37.453 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pg_log_dups_tracked": "3000", 2023-12-15T22:56:37.453 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pg_log_trim_max": "10000", 2023-12-15T22:56:37.454 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pg_log_trim_min": "100", 2023-12-15T22:56:37.454 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2023-12-15T22:56:37.454 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pg_object_context_cache_count": "64", 2023-12-15T22:56:37.454 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pg_stat_report_interval_max": "500", 2023-12-15T22:56:37.454 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2023-12-15T22:56:37.454 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pool_default_cache_min_evict_age": "0", 2023-12-15T22:56:37.454 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pool_default_cache_min_flush_age": "0", 2023-12-15T22:56:37.454 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2023-12-15T22:56:37.454 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2023-12-15T22:56:37.454 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2023-12-15T22:56:37.454 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pool_default_crimson": "true", 2023-12-15T22:56:37.454 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pool_default_crush_rule": "-1", 2023-12-15T22:56:37.454 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pool_default_ec_fast_read": "false", 2023-12-15T22:56:37.454 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2023-12-15T22:56:37.454 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pool_default_flag_bulk": "false", 2023-12-15T22:56:37.455 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pool_default_flag_hashpspool": "true", 2023-12-15T22:56:37.455 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pool_default_flag_nodelete": "false", 2023-12-15T22:56:37.455 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pool_default_flag_nopgchange": "false", 2023-12-15T22:56:37.455 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pool_default_flag_nosizechange": "false", 2023-12-15T22:56:37.455 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pool_default_flags": "0", 2023-12-15T22:56:37.455 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2023-12-15T22:56:37.455 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pool_default_min_size": "2", 2023-12-15T22:56:37.455 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2023-12-15T22:56:37.455 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pool_default_pg_num": "32", 2023-12-15T22:56:37.455 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pool_default_pgp_num": "0", 2023-12-15T22:56:37.455 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2023-12-15T22:56:37.455 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pool_default_size": "3", 2023-12-15T22:56:37.455 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pool_default_type": "replicated", 2023-12-15T22:56:37.455 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2023-12-15T22:56:37.455 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pool_use_gmt_hitset": "true", 2023-12-15T22:56:37.456 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_push_per_object_cost": "1000", 2023-12-15T22:56:37.456 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_read_ec_check_for_errors": "false", 2023-12-15T22:56:37.456 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_recover_clone_overlap": "true", 2023-12-15T22:56:37.456 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_recover_clone_overlap_limit": "10", 2023-12-15T22:56:37.456 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_recovery_cost": "20971520", 2023-12-15T22:56:37.456 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_recovery_delay_start": "0.000000", 2023-12-15T22:56:37.456 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_recovery_max_active": "0", 2023-12-15T22:56:37.456 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_recovery_max_active_hdd": "3", 2023-12-15T22:56:37.456 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_recovery_max_active_ssd": "10", 2023-12-15T22:56:37.456 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_recovery_max_chunk": "8388608", 2023-12-15T22:56:37.456 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2023-12-15T22:56:37.456 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_recovery_max_single_start": "1", 2023-12-15T22:56:37.456 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_recovery_op_priority": "3", 2023-12-15T22:56:37.456 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_recovery_op_warn_multiple": "16", 2023-12-15T22:56:37.456 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_recovery_priority": "5", 2023-12-15T22:56:37.457 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_recovery_retry_interval": "30.000000", 2023-12-15T22:56:37.457 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_recovery_sleep": "0.000000", 2023-12-15T22:56:37.457 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_recovery_sleep_hdd": "0.100000", 2023-12-15T22:56:37.457 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2023-12-15T22:56:37.457 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_recovery_sleep_ssd": "0.000000", 2023-12-15T22:56:37.457 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_repair_during_recovery": "false", 2023-12-15T22:56:37.457 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_requested_scrub_priority": "120", 2023-12-15T22:56:37.457 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2023-12-15T22:56:37.457 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_rollback_to_cluster_snap": "", 2023-12-15T22:56:37.457 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_scrub_auto_repair": "false", 2023-12-15T22:56:37.457 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_scrub_auto_repair_num_errors": "5", 2023-12-15T22:56:37.457 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_scrub_backoff_ratio": "0.660000", 2023-12-15T22:56:37.457 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_scrub_begin_hour": "0", 2023-12-15T22:56:37.457 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_scrub_begin_week_day": "0", 2023-12-15T22:56:37.458 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_scrub_chunk_max": "25", 2023-12-15T22:56:37.458 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_scrub_chunk_min": "5", 2023-12-15T22:56:37.458 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_scrub_cost": "52428800", 2023-12-15T22:56:37.458 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_scrub_during_recovery": "false", 2023-12-15T22:56:37.458 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_scrub_end_hour": "0", 2023-12-15T22:56:37.458 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_scrub_end_week_day": "0", 2023-12-15T22:56:37.458 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_scrub_event_cost": "4096", 2023-12-15T22:56:37.458 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_scrub_extended_sleep": "0.000000", 2023-12-15T22:56:37.458 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2023-12-15T22:56:37.458 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_scrub_invalid_stats": "true", 2023-12-15T22:56:37.458 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_scrub_load_threshold": "0.500000", 2023-12-15T22:56:37.458 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_scrub_max_interval": "604800.000000", 2023-12-15T22:56:37.458 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_scrub_max_preemptions": "5", 2023-12-15T22:56:37.458 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_scrub_min_interval": "86400.000000", 2023-12-15T22:56:37.458 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_scrub_priority": "5", 2023-12-15T22:56:37.459 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_scrub_reservation_timeout": "300000", 2023-12-15T22:56:37.459 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_scrub_sleep": "0.000000", 2023-12-15T22:56:37.459 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_scrub_slow_reservation_response": "30000", 2023-12-15T22:56:37.459 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_shallow_scrub_chunk_max": "100", 2023-12-15T22:56:37.459 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_shallow_scrub_chunk_min": "50", 2023-12-15T22:56:37.459 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_shutdown_pgref_assert": "false", 2023-12-15T22:56:37.459 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_skip_data_digest": "false", 2023-12-15T22:56:37.459 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_smart_report_timeout": "5", 2023-12-15T22:56:37.459 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_snap_trim_cost": "1048576", 2023-12-15T22:56:37.459 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_snap_trim_priority": "5", 2023-12-15T22:56:37.459 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_snap_trim_sleep": "0.000000", 2023-12-15T22:56:37.459 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2023-12-15T22:56:37.459 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2023-12-15T22:56:37.459 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2023-12-15T22:56:37.460 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_stats_update_period_not_scrubbing": "120", 2023-12-15T22:56:37.460 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_stats_update_period_scrubbing": "15", 2023-12-15T22:56:37.460 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2023-12-15T22:56:37.460 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_target_transaction_size": "30", 2023-12-15T22:56:37.460 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_tier_default_cache_hit_set_count": "4", 2023-12-15T22:56:37.460 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2023-12-15T22:56:37.460 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2023-12-15T22:56:37.460 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2023-12-15T22:56:37.460 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2023-12-15T22:56:37.460 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2023-12-15T22:56:37.460 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2023-12-15T22:56:37.460 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_tier_default_cache_mode": "writeback", 2023-12-15T22:56:37.460 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2023-12-15T22:56:37.460 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_tier_promote_max_objects_sec": "25", 2023-12-15T22:56:37.460 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_tracing": "false", 2023-12-15T22:56:37.461 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_use_stale_snap": "false", 2023-12-15T22:56:37.461 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2023-12-15T22:56:37.461 INFO:teuthology.orchestra.run.smithi071.stdout: "osdc_blkin_trace_all": "false", 2023-12-15T22:56:37.461 INFO:teuthology.orchestra.run.smithi071.stdout: "paxos_kill_at": "0", 2023-12-15T22:56:37.461 INFO:teuthology.orchestra.run.smithi071.stdout: "paxos_max_join_drift": "10", 2023-12-15T22:56:37.461 INFO:teuthology.orchestra.run.smithi071.stdout: "paxos_min": "500", 2023-12-15T22:56:37.461 INFO:teuthology.orchestra.run.smithi071.stdout: "paxos_min_wait": "0.050000", 2023-12-15T22:56:37.461 INFO:teuthology.orchestra.run.smithi071.stdout: "paxos_propose_interval": "1.000000", 2023-12-15T22:56:37.461 INFO:teuthology.orchestra.run.smithi071.stdout: "paxos_service_trim_max": "500", 2023-12-15T22:56:37.461 INFO:teuthology.orchestra.run.smithi071.stdout: "paxos_service_trim_max_multiplier": "20", 2023-12-15T22:56:37.461 INFO:teuthology.orchestra.run.smithi071.stdout: "paxos_service_trim_min": "10", 2023-12-15T22:56:37.461 INFO:teuthology.orchestra.run.smithi071.stdout: "paxos_stash_full_interval": "25", 2023-12-15T22:56:37.461 INFO:teuthology.orchestra.run.smithi071.stdout: "paxos_trim_max": "500", 2023-12-15T22:56:37.461 INFO:teuthology.orchestra.run.smithi071.stdout: "paxos_trim_min": "250", 2023-12-15T22:56:37.462 INFO:teuthology.orchestra.run.smithi071.stdout: "perf": "true", 2023-12-15T22:56:37.462 INFO:teuthology.orchestra.run.smithi071.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2023-12-15T22:56:37.462 INFO:teuthology.orchestra.run.smithi071.stdout: "plugin_crypto_accelerator": "crypto_isal", 2023-12-15T22:56:37.462 INFO:teuthology.orchestra.run.smithi071.stdout: "plugin_dir": "/usr/lib64/ceph", 2023-12-15T22:56:37.462 INFO:teuthology.orchestra.run.smithi071.stdout: "public_addr": "-", 2023-12-15T22:56:37.462 INFO:teuthology.orchestra.run.smithi071.stdout: "public_addrv": "", 2023-12-15T22:56:37.462 INFO:teuthology.orchestra.run.smithi071.stdout: "public_bind_addr": "-", 2023-12-15T22:56:37.462 INFO:teuthology.orchestra.run.smithi071.stdout: "public_network": "", 2023-12-15T22:56:37.462 INFO:teuthology.orchestra.run.smithi071.stdout: "public_network_interface": "", 2023-12-15T22:56:37.462 INFO:teuthology.orchestra.run.smithi071.stdout: "qat_compressor_enabled": "false", 2023-12-15T22:56:37.462 INFO:teuthology.orchestra.run.smithi071.stdout: "qat_compressor_session_max_number": "256", 2023-12-15T22:56:37.462 INFO:teuthology.orchestra.run.smithi071.stdout: "rados_mon_op_timeout": "0", 2023-12-15T22:56:37.462 INFO:teuthology.orchestra.run.smithi071.stdout: "rados_osd_op_timeout": "0", 2023-12-15T22:56:37.462 INFO:teuthology.orchestra.run.smithi071.stdout: "rados_tracing": "false", 2023-12-15T22:56:37.462 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_atime_update_interval": "60", 2023-12-15T22:56:37.463 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2023-12-15T22:56:37.463 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_balance_parent_reads": "false", 2023-12-15T22:56:37.463 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_balance_snap_reads": "false", 2023-12-15T22:56:37.463 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_blkin_trace_all": "false", 2023-12-15T22:56:37.463 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_blocklist_expire_seconds": "0", 2023-12-15T22:56:37.463 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_blocklist_on_break_lock": "true", 2023-12-15T22:56:37.463 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_cache": "true", 2023-12-15T22:56:37.463 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_cache_block_writes_upfront": "false", 2023-12-15T22:56:37.463 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_cache_max_dirty": "25165824", 2023-12-15T22:56:37.463 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_cache_max_dirty_age": "1.000000", 2023-12-15T22:56:37.463 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_cache_max_dirty_object": "0", 2023-12-15T22:56:37.463 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_cache_policy": "writearound", 2023-12-15T22:56:37.463 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_cache_size": "33554432", 2023-12-15T22:56:37.463 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_cache_target_dirty": "16777216", 2023-12-15T22:56:37.463 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_cache_writethrough_until_flush": "true", 2023-12-15T22:56:37.464 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_clone_copy_on_read": "false", 2023-12-15T22:56:37.464 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_compression_hint": "none", 2023-12-15T22:56:37.464 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_concurrent_management_ops": "10", 2023-12-15T22:56:37.464 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_config_pool_override_update_timestamp": "0", 2023-12-15T22:56:37.464 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_default_clone_format": "auto", 2023-12-15T22:56:37.464 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_default_data_pool": "", 2023-12-15T22:56:37.464 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_default_features": "61", 2023-12-15T22:56:37.464 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_default_format": "2", 2023-12-15T22:56:37.464 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_default_map_options": "", 2023-12-15T22:56:37.464 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_default_order": "22", 2023-12-15T22:56:37.464 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_default_pool": "rbd", 2023-12-15T22:56:37.464 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2023-12-15T22:56:37.464 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_default_stripe_count": "0", 2023-12-15T22:56:37.464 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_default_stripe_unit": "0", 2023-12-15T22:56:37.464 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_disable_zero_copy_writes": "true", 2023-12-15T22:56:37.465 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_discard_granularity_bytes": "65536", 2023-12-15T22:56:37.465 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_discard_on_zeroed_write_same": "true", 2023-12-15T22:56:37.465 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_enable_alloc_hint": "true", 2023-12-15T22:56:37.465 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2023-12-15T22:56:37.465 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_io_scheduler": "simple", 2023-12-15T22:56:37.465 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2023-12-15T22:56:37.465 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_journal_commit_age": "5.000000", 2023-12-15T22:56:37.465 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2023-12-15T22:56:37.465 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_journal_max_payload_bytes": "16384", 2023-12-15T22:56:37.465 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_journal_object_flush_age": "0.000000", 2023-12-15T22:56:37.465 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_journal_object_flush_bytes": "1048576", 2023-12-15T22:56:37.465 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_journal_object_flush_interval": "0", 2023-12-15T22:56:37.465 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2023-12-15T22:56:37.465 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2023-12-15T22:56:37.466 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_journal_order": "24", 2023-12-15T22:56:37.466 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_journal_pool": "", 2023-12-15T22:56:37.466 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_journal_splay_width": "4", 2023-12-15T22:56:37.466 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_localize_parent_reads": "false", 2023-12-15T22:56:37.466 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_localize_snap_reads": "false", 2023-12-15T22:56:37.466 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2023-12-15T22:56:37.466 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2023-12-15T22:56:37.466 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2023-12-15T22:56:37.466 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2023-12-15T22:56:37.466 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2023-12-15T22:56:37.466 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2023-12-15T22:56:37.466 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_mirror_image_policy_type": "simple", 2023-12-15T22:56:37.466 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2023-12-15T22:56:37.466 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_mirror_image_state_check_interval": "30", 2023-12-15T22:56:37.467 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2023-12-15T22:56:37.467 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2023-12-15T22:56:37.467 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2023-12-15T22:56:37.467 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2023-12-15T22:56:37.467 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2023-12-15T22:56:37.467 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_mirror_memory_autotune": "true", 2023-12-15T22:56:37.467 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_mirror_memory_base": "805306368", 2023-12-15T22:56:37.467 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2023-12-15T22:56:37.467 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_mirror_memory_cache_min": "134217728", 2023-12-15T22:56:37.467 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2023-12-15T22:56:37.467 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2023-12-15T22:56:37.467 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_mirror_memory_target": "4294967296", 2023-12-15T22:56:37.467 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_mirror_perf_stats_prio": "5", 2023-12-15T22:56:37.467 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2023-12-15T22:56:37.467 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2023-12-15T22:56:37.468 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_mirroring_delete_delay": "0", 2023-12-15T22:56:37.468 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2023-12-15T22:56:37.468 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_mirroring_replay_delay": "0", 2023-12-15T22:56:37.468 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2023-12-15T22:56:37.468 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2023-12-15T22:56:37.468 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_move_to_trash_on_remove": "false", 2023-12-15T22:56:37.468 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2023-12-15T22:56:37.468 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_mtime_update_interval": "60", 2023-12-15T22:56:37.468 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_non_blocking_aio": "true", 2023-12-15T22:56:37.468 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_op_thread_timeout": "60", 2023-12-15T22:56:37.468 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_op_threads": "1", 2023-12-15T22:56:37.468 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_parent_cache_enabled": "false", 2023-12-15T22:56:37.468 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_persistent_cache_mode": "disabled", 2023-12-15T22:56:37.468 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_persistent_cache_path": "/tmp", 2023-12-15T22:56:37.469 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_persistent_cache_size": "1073741824", 2023-12-15T22:56:37.469 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_plugins": "", 2023-12-15T22:56:37.469 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_qos_bps_burst": "0", 2023-12-15T22:56:37.469 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_qos_bps_burst_seconds": "1", 2023-12-15T22:56:37.469 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_qos_bps_limit": "0", 2023-12-15T22:56:37.469 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_qos_exclude_ops": "0", 2023-12-15T22:56:37.469 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_qos_iops_burst": "0", 2023-12-15T22:56:37.469 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_qos_iops_burst_seconds": "1", 2023-12-15T22:56:37.469 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_qos_iops_limit": "0", 2023-12-15T22:56:37.469 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_qos_read_bps_burst": "0", 2023-12-15T22:56:37.469 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2023-12-15T22:56:37.469 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_qos_read_bps_limit": "0", 2023-12-15T22:56:37.469 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_qos_read_iops_burst": "0", 2023-12-15T22:56:37.469 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2023-12-15T22:56:37.469 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_qos_read_iops_limit": "0", 2023-12-15T22:56:37.470 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_qos_schedule_tick_min": "50", 2023-12-15T22:56:37.470 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_qos_write_bps_burst": "0", 2023-12-15T22:56:37.470 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2023-12-15T22:56:37.470 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_qos_write_bps_limit": "0", 2023-12-15T22:56:37.470 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_qos_write_iops_burst": "0", 2023-12-15T22:56:37.470 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2023-12-15T22:56:37.470 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_qos_write_iops_limit": "0", 2023-12-15T22:56:37.470 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_quiesce_notification_attempts": "10", 2023-12-15T22:56:37.470 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_read_from_replica_policy": "default", 2023-12-15T22:56:37.470 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2023-12-15T22:56:37.470 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_readahead_max_bytes": "524288", 2023-12-15T22:56:37.470 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_readahead_trigger_requests": "10", 2023-12-15T22:56:37.470 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_request_timed_out_seconds": "30", 2023-12-15T22:56:37.470 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_skip_partial_discard": "true", 2023-12-15T22:56:37.471 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2023-12-15T22:56:37.471 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_tracing": "false", 2023-12-15T22:56:37.471 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_validate_names": "true", 2023-12-15T22:56:37.471 INFO:teuthology.orchestra.run.smithi071.stdout: "rbd_validate_pool": "true", 2023-12-15T22:56:37.471 INFO:teuthology.orchestra.run.smithi071.stdout: "restapi_base_url": "", 2023-12-15T22:56:37.471 INFO:teuthology.orchestra.run.smithi071.stdout: "restapi_log_level": "", 2023-12-15T22:56:37.471 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_acl_grants_max_num": "100", 2023-12-15T22:56:37.471 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_admin_entry": "admin", 2023-12-15T22:56:37.471 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2023-12-15T22:56:37.471 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_backend_store": "rados", 2023-12-15T22:56:37.471 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_barbican_url": "", 2023-12-15T22:56:37.471 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_beast_enable_async": "true", 2023-12-15T22:56:37.471 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_bucket_counters_cache": "false", 2023-12-15T22:56:37.471 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_bucket_counters_cache_size": "10000", 2023-12-15T22:56:37.472 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2023-12-15T22:56:37.472 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_bucket_default_quota_max_size": "-1", 2023-12-15T22:56:37.472 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_bucket_index_max_aio": "128", 2023-12-15T22:56:37.472 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2023-12-15T22:56:37.472 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_bucket_quota_cache_size": "10000", 2023-12-15T22:56:37.472 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_bucket_quota_ttl": "600", 2023-12-15T22:56:37.472 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_bucket_sync_spawn_window": "20", 2023-12-15T22:56:37.472 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_cache_enabled": "true", 2023-12-15T22:56:37.472 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_cache_expiry_interval": "900", 2023-12-15T22:56:37.472 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_cache_lru_size": "10000", 2023-12-15T22:56:37.472 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_config_store": "rados", 2023-12-15T22:56:37.472 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_content_length_compat": "false", 2023-12-15T22:56:37.472 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_copy_obj_progress": "true", 2023-12-15T22:56:37.472 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2023-12-15T22:56:37.472 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_cors_rules_max_num": "100", 2023-12-15T22:56:37.473 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_cross_domain_policy": "", 2023-12-15T22:56:37.473 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_crypt_default_encryption_key": "", 2023-12-15T22:56:37.473 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_crypt_kmip_addr": "", 2023-12-15T22:56:37.473 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_crypt_kmip_ca_path": "", 2023-12-15T22:56:37.473 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_crypt_kmip_client_cert": "", 2023-12-15T22:56:37.473 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_crypt_kmip_client_key": "", 2023-12-15T22:56:37.473 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_crypt_kmip_kms_key_template": "", 2023-12-15T22:56:37.473 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_crypt_kmip_password": "", 2023-12-15T22:56:37.473 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2023-12-15T22:56:37.473 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_crypt_kmip_username": "", 2023-12-15T22:56:37.473 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_crypt_require_ssl": "true", 2023-12-15T22:56:37.473 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2023-12-15T22:56:37.473 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2023-12-15T22:56:37.473 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_crypt_sse_s3_backend": "vault", 2023-12-15T22:56:37.474 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2023-12-15T22:56:37.474 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2023-12-15T22:56:37.474 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2023-12-15T22:56:37.474 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2023-12-15T22:56:37.474 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2023-12-15T22:56:37.474 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2023-12-15T22:56:37.474 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2023-12-15T22:56:37.474 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2023-12-15T22:56:37.474 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2023-12-15T22:56:37.474 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2023-12-15T22:56:37.474 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2023-12-15T22:56:37.474 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_crypt_suppress_logs": "true", 2023-12-15T22:56:37.474 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_crypt_vault_addr": "", 2023-12-15T22:56:37.474 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_crypt_vault_auth": "token", 2023-12-15T22:56:37.475 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_crypt_vault_namespace": "", 2023-12-15T22:56:37.475 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_crypt_vault_prefix": "", 2023-12-15T22:56:37.475 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_crypt_vault_secret_engine": "transit", 2023-12-15T22:56:37.475 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_crypt_vault_ssl_cacert": "", 2023-12-15T22:56:37.475 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2023-12-15T22:56:37.475 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2023-12-15T22:56:37.475 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_crypt_vault_token_file": "", 2023-12-15T22:56:37.475 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_crypt_vault_verify_ssl": "true", 2023-12-15T22:56:37.475 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_curl_buffersize": "524288", 2023-12-15T22:56:37.475 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_curl_low_speed_limit": "1024", 2023-12-15T22:56:37.475 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_curl_low_speed_time": "300", 2023-12-15T22:56:37.475 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_curl_tcp_keepalive": "0", 2023-12-15T22:56:37.475 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_curl_wait_timeout_ms": "1000", 2023-12-15T22:56:37.475 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2023-12-15T22:56:37.475 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2023-12-15T22:56:37.476 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2023-12-15T22:56:37.476 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2023-12-15T22:56:37.476 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_d3n_l1_fadvise": "4", 2023-12-15T22:56:37.476 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2023-12-15T22:56:37.476 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_d3n_libaio_aio_num": "64", 2023-12-15T22:56:37.476 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_d3n_libaio_aio_threads": "20", 2023-12-15T22:56:37.476 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_d4n_host": "127.0.0.1", 2023-12-15T22:56:37.476 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_d4n_port": "6379", 2023-12-15T22:56:37.476 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2023-12-15T22:56:37.476 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_data_log_changes_size": "1000", 2023-12-15T22:56:37.476 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_data_log_num_shards": "128", 2023-12-15T22:56:37.476 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_data_log_obj_prefix": "data_log", 2023-12-15T22:56:37.476 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_data_log_window": "30", 2023-12-15T22:56:37.476 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_data_notify_interval_msec": "0", 2023-12-15T22:56:37.476 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_data_sync_poll_interval": "20", 2023-12-15T22:56:37.477 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_data_sync_spawn_window": "20", 2023-12-15T22:56:37.477 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2023-12-15T22:56:37.477 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_debug_inject_set_olh_err": "0", 2023-12-15T22:56:37.477 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_default_data_log_backing": "fifo", 2023-12-15T22:56:37.477 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_default_realm_info_oid": "default.realm", 2023-12-15T22:56:37.477 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_default_region_info_oid": "default.region", 2023-12-15T22:56:37.477 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_default_zone_info_oid": "default.zone", 2023-12-15T22:56:37.477 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2023-12-15T22:56:37.477 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_defer_to_bucket_acls": "", 2023-12-15T22:56:37.477 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_delete_multi_obj_max_num": "1000", 2023-12-15T22:56:37.477 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_dmclock_admin_lim": "0.000000", 2023-12-15T22:56:37.477 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_dmclock_admin_res": "100.000000", 2023-12-15T22:56:37.477 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2023-12-15T22:56:37.477 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_dmclock_auth_lim": "0.000000", 2023-12-15T22:56:37.478 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_dmclock_auth_res": "200.000000", 2023-12-15T22:56:37.478 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2023-12-15T22:56:37.478 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_dmclock_data_lim": "0.000000", 2023-12-15T22:56:37.478 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_dmclock_data_res": "500.000000", 2023-12-15T22:56:37.478 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_dmclock_data_wgt": "500.000000", 2023-12-15T22:56:37.478 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2023-12-15T22:56:37.478 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_dmclock_metadata_res": "500.000000", 2023-12-15T22:56:37.478 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2023-12-15T22:56:37.478 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_dns_name": "", 2023-12-15T22:56:37.478 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_dns_s3website_name": "", 2023-12-15T22:56:37.478 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_dynamic_resharding": "true", 2023-12-15T22:56:37.478 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2023-12-15T22:56:37.478 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_enable_gc_threads": "true", 2023-12-15T22:56:37.478 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_enable_lc_threads": "true", 2023-12-15T22:56:37.478 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_enable_ops_log": "false", 2023-12-15T22:56:37.479 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_enable_quota_threads": "true", 2023-12-15T22:56:37.479 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_enable_static_website": "false", 2023-12-15T22:56:37.479 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_enable_usage_log": "false", 2023-12-15T22:56:37.479 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_enforce_swift_acls": "true", 2023-12-15T22:56:37.479 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_exit_timeout_secs": "120", 2023-12-15T22:56:37.479 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_expose_bucket": "false", 2023-12-15T22:56:37.479 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_extended_http_attrs": "", 2023-12-15T22:56:37.479 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_filter": "none", 2023-12-15T22:56:37.479 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2023-12-15T22:56:37.479 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_frontends": "beast port=7480", 2023-12-15T22:56:37.479 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_gc_max_concurrent_io": "10", 2023-12-15T22:56:37.479 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_gc_max_deferred": "50", 2023-12-15T22:56:37.479 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2023-12-15T22:56:37.479 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_gc_max_objs": "32", 2023-12-15T22:56:37.480 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_gc_max_queue_size": "134213632", 2023-12-15T22:56:37.480 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_gc_max_trim_chunk": "16", 2023-12-15T22:56:37.480 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_gc_obj_min_wait": "7200", 2023-12-15T22:56:37.480 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_gc_processor_max_time": "3600", 2023-12-15T22:56:37.480 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_gc_processor_period": "3600", 2023-12-15T22:56:37.480 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_get_obj_max_req_size": "4194304", 2023-12-15T22:56:37.480 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_get_obj_window_size": "16777216", 2023-12-15T22:56:37.480 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_healthcheck_disabling_path": "", 2023-12-15T22:56:37.480 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_ignore_get_invalid_range": "false", 2023-12-15T22:56:37.480 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_init_timeout": "300", 2023-12-15T22:56:37.480 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2023-12-15T22:56:37.480 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2023-12-15T22:56:37.480 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_keystone_accepted_admin_roles": "", 2023-12-15T22:56:37.480 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_keystone_accepted_reader_roles": "", 2023-12-15T22:56:37.480 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2023-12-15T22:56:37.481 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_keystone_admin_domain": "", 2023-12-15T22:56:37.481 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_keystone_admin_password": "", 2023-12-15T22:56:37.481 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_keystone_admin_password_path": "", 2023-12-15T22:56:37.481 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_keystone_admin_project": "", 2023-12-15T22:56:37.481 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_keystone_admin_tenant": "", 2023-12-15T22:56:37.481 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_keystone_admin_token": "", 2023-12-15T22:56:37.481 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_keystone_admin_token_path": "", 2023-12-15T22:56:37.481 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_keystone_admin_user": "", 2023-12-15T22:56:37.481 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_keystone_api_version": "2", 2023-12-15T22:56:37.481 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_keystone_barbican_domain": "", 2023-12-15T22:56:37.481 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_keystone_barbican_password": "", 2023-12-15T22:56:37.481 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_keystone_barbican_project": "", 2023-12-15T22:56:37.481 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_keystone_barbican_tenant": "", 2023-12-15T22:56:37.482 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_keystone_barbican_user": "", 2023-12-15T22:56:37.482 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2023-12-15T22:56:37.482 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_keystone_implicit_tenants": "false", 2023-12-15T22:56:37.482 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2023-12-15T22:56:37.482 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_keystone_service_token_enabled": "false", 2023-12-15T22:56:37.482 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_keystone_token_cache_size": "10000", 2023-12-15T22:56:37.482 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_keystone_url": "", 2023-12-15T22:56:37.482 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_keystone_verify_ssl": "true", 2023-12-15T22:56:37.482 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_lc_debug_interval": "-1", 2023-12-15T22:56:37.482 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_lc_lock_max_time": "90", 2023-12-15T22:56:37.482 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_lc_max_objs": "32", 2023-12-15T22:56:37.482 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_lc_max_rules": "1000", 2023-12-15T22:56:37.482 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_lc_max_worker": "3", 2023-12-15T22:56:37.482 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_lc_max_wp_worker": "3", 2023-12-15T22:56:37.482 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_lc_thread_delay": "0", 2023-12-15T22:56:37.483 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2023-12-15T22:56:37.483 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_ldap_dnattr": "uid", 2023-12-15T22:56:37.483 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2023-12-15T22:56:37.483 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_ldap_searchfilter": "", 2023-12-15T22:56:37.483 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2023-12-15T22:56:37.483 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_ldap_uri": "ldaps://", 2023-12-15T22:56:37.483 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2023-12-15T22:56:37.483 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_list_bucket_min_readahead": "1000", 2023-12-15T22:56:37.483 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_list_buckets_max_chunk": "1000", 2023-12-15T22:56:37.483 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_log_http_headers": "", 2023-12-15T22:56:37.483 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_log_nonexistent_bucket": "false", 2023-12-15T22:56:37.483 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2023-12-15T22:56:37.483 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_log_object_name_utc": "false", 2023-12-15T22:56:37.483 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_lua_max_memory_per_state": "128000", 2023-12-15T22:56:37.484 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2023-12-15T22:56:37.484 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_max_attr_name_len": "0", 2023-12-15T22:56:37.484 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_max_attr_size": "0", 2023-12-15T22:56:37.484 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_max_attrs_num_in_req": "0", 2023-12-15T22:56:37.484 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_max_chunk_size": "4194304", 2023-12-15T22:56:37.484 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_max_concurrent_requests": "1024", 2023-12-15T22:56:37.484 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2023-12-15T22:56:37.484 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_max_dynamic_shards": "1999", 2023-12-15T22:56:37.484 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_max_listing_results": "1000", 2023-12-15T22:56:37.484 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_max_notify_retries": "10", 2023-12-15T22:56:37.484 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_max_objs_per_shard": "100000", 2023-12-15T22:56:37.484 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_max_put_param_size": "1048576", 2023-12-15T22:56:37.484 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_max_put_size": "5368709120", 2023-12-15T22:56:37.484 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_max_slo_entries": "1000", 2023-12-15T22:56:37.485 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_md_log_max_shards": "64", 2023-12-15T22:56:37.485 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_md_notify_interval_msec": "200", 2023-12-15T22:56:37.485 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_meta_sync_poll_interval": "20", 2023-12-15T22:56:37.485 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_meta_sync_spawn_window": "20", 2023-12-15T22:56:37.485 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_mime_types_file": "/etc/mime.types", 2023-12-15T22:56:37.485 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_mp_lock_max_time": "600", 2023-12-15T22:56:37.485 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_multi_obj_del_max_aio": "16", 2023-12-15T22:56:37.485 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_multipart_min_part_size": "5242880", 2023-12-15T22:56:37.485 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_multipart_part_upload_limit": "10000", 2023-12-15T22:56:37.485 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_nfs_fhcache_partitions": "3", 2023-12-15T22:56:37.485 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_nfs_fhcache_size": "2017", 2023-12-15T22:56:37.485 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_nfs_frontends": "rgw-nfs", 2023-12-15T22:56:37.485 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2023-12-15T22:56:37.485 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_nfs_lru_lanes": "5", 2023-12-15T22:56:37.486 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_nfs_max_gc": "300", 2023-12-15T22:56:37.486 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_nfs_namespace_expire_secs": "300", 2023-12-15T22:56:37.486 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_nfs_run_gc_threads": "false", 2023-12-15T22:56:37.486 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_nfs_run_lc_threads": "false", 2023-12-15T22:56:37.486 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_nfs_run_quota_threads": "false", 2023-12-15T22:56:37.486 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_nfs_run_sync_thread": "false", 2023-12-15T22:56:37.486 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_nfs_s3_fast_attrs": "false", 2023-12-15T22:56:37.486 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_nfs_write_completion_interval_s": "10", 2023-12-15T22:56:37.486 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_num_async_rados_threads": "32", 2023-12-15T22:56:37.486 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_num_control_oids": "8", 2023-12-15T22:56:37.486 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_numa_node": "-1", 2023-12-15T22:56:37.486 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_obj_stripe_size": "4194304", 2023-12-15T22:56:37.486 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_obj_tombstone_cache_size": "1000", 2023-12-15T22:56:37.486 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_objexp_chunk_size": "100", 2023-12-15T22:56:37.487 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_objexp_gc_interval": "600", 2023-12-15T22:56:37.487 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_objexp_hints_num_shards": "127", 2023-12-15T22:56:37.487 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_olh_pending_timeout_sec": "3600", 2023-12-15T22:56:37.487 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_op_thread_suicide_timeout": "0", 2023-12-15T22:56:37.487 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_op_thread_timeout": "600", 2023-12-15T22:56:37.487 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_op_tracing": "false", 2023-12-15T22:56:37.487 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_opa_token": "", 2023-12-15T22:56:37.487 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_opa_url": "", 2023-12-15T22:56:37.487 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_opa_verify_ssl": "true", 2023-12-15T22:56:37.487 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_ops_log_data_backlog": "5242880", 2023-12-15T22:56:37.487 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2023-12-15T22:56:37.487 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_ops_log_rados": "false", 2023-12-15T22:56:37.487 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_ops_log_socket_path": "", 2023-12-15T22:56:37.487 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_override_bucket_index_max_shards": "0", 2023-12-15T22:56:37.487 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2023-12-15T22:56:37.488 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2023-12-15T22:56:37.488 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_period_push_interval": "2.000000", 2023-12-15T22:56:37.488 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_period_push_interval_max": "30.000000", 2023-12-15T22:56:37.488 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_period_root_pool": ".rgw.root", 2023-12-15T22:56:37.488 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_policy_reject_invalid_principals": "true", 2023-12-15T22:56:37.488 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2023-12-15T22:56:37.488 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_posix_cache_lanes": "3", 2023-12-15T22:56:37.488 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_posix_cache_lmdb_count": "3", 2023-12-15T22:56:37.488 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_posix_cache_max_buckets": "100", 2023-12-15T22:56:37.488 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_posix_cache_partitions": "3", 2023-12-15T22:56:37.488 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2023-12-15T22:56:37.488 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_print_continue": "true", 2023-12-15T22:56:37.488 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_print_prohibited_content_length": "false", 2023-12-15T22:56:37.488 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_put_obj_max_window_size": "67108864", 2023-12-15T22:56:37.488 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_put_obj_min_window_size": "16777216", 2023-12-15T22:56:37.489 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2023-12-15T22:56:37.489 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_rados_pool_recovery_priority": "5", 2023-12-15T22:56:37.489 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_rados_tracing": "false", 2023-12-15T22:56:37.489 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_realm": "", 2023-12-15T22:56:37.489 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_realm_id": "", 2023-12-15T22:56:37.489 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_realm_root_pool": ".rgw.root", 2023-12-15T22:56:37.489 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_region": "", 2023-12-15T22:56:37.489 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_region_root_pool": ".rgw.root", 2023-12-15T22:56:37.489 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_relaxed_region_enforcement": "false", 2023-12-15T22:56:37.489 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_relaxed_s3_bucket_names": "false", 2023-12-15T22:56:37.489 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2023-12-15T22:56:37.489 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_request_uri": "", 2023-12-15T22:56:37.489 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_reshard_batch_size": "64", 2023-12-15T22:56:37.489 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_reshard_bucket_lock_duration": "360", 2023-12-15T22:56:37.490 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_reshard_max_aio": "128", 2023-12-15T22:56:37.490 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_reshard_num_logs": "16", 2023-12-15T22:56:37.490 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_reshard_thread_interval": "600", 2023-12-15T22:56:37.490 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_resolve_cname": "false", 2023-12-15T22:56:37.490 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_rest_getusage_op_compat": "false", 2023-12-15T22:56:37.490 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_run_sync_thread": "true", 2023-12-15T22:56:37.490 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_s3_auth_order": "sts, external, local", 2023-12-15T22:56:37.490 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_s3_auth_use_keystone": "false", 2023-12-15T22:56:37.490 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_s3_auth_use_ldap": "false", 2023-12-15T22:56:37.490 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_s3_auth_use_rados": "true", 2023-12-15T22:56:37.490 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_s3_auth_use_sts": "false", 2023-12-15T22:56:37.490 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_s3_client_max_sig_ver": "-1", 2023-12-15T22:56:37.490 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_s3_success_create_obj_status": "0", 2023-12-15T22:56:37.490 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_safe_max_objects_per_shard": "102400", 2023-12-15T22:56:37.491 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_scheduler_type": "throttler", 2023-12-15T22:56:37.491 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_script_uri": "", 2023-12-15T22:56:37.491 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_service_provider_name": "", 2023-12-15T22:56:37.491 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_shard_warning_threshold": "90.000000", 2023-12-15T22:56:37.491 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_sts_client_id": "", 2023-12-15T22:56:37.491 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_sts_client_secret": "", 2023-12-15T22:56:37.491 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_sts_entry": "sts", 2023-12-15T22:56:37.491 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_sts_key": "sts", 2023-12-15T22:56:37.491 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_sts_max_session_duration": "43200", 2023-12-15T22:56:37.491 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_sts_min_session_duration": "900", 2023-12-15T22:56:37.491 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_sts_token_introspection_url": "", 2023-12-15T22:56:37.491 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_swift_account_in_url": "false", 2023-12-15T22:56:37.491 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_swift_auth_entry": "auth", 2023-12-15T22:56:37.491 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_swift_auth_url": "", 2023-12-15T22:56:37.491 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_swift_custom_header": "", 2023-12-15T22:56:37.492 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_swift_enforce_content_length": "false", 2023-12-15T22:56:37.492 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_swift_need_stats": "true", 2023-12-15T22:56:37.492 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_swift_tenant_name": "", 2023-12-15T22:56:37.492 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_swift_token_expiration": "86400", 2023-12-15T22:56:37.492 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_swift_url": "", 2023-12-15T22:56:37.492 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_swift_url_prefix": "swift", 2023-12-15T22:56:37.492 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_swift_versioning_enabled": "false", 2023-12-15T22:56:37.492 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2023-12-15T22:56:37.492 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2023-12-15T22:56:37.492 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_sync_lease_period": "120", 2023-12-15T22:56:37.492 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2023-12-15T22:56:37.492 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_sync_log_trim_interval": "1200", 2023-12-15T22:56:37.492 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_sync_log_trim_max_buckets": "16", 2023-12-15T22:56:37.492 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2023-12-15T22:56:37.493 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2023-12-15T22:56:37.493 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_sync_obj_etag_verify": "false", 2023-12-15T22:56:37.493 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_sync_trace_history_size": "4096", 2023-12-15T22:56:37.493 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_sync_trace_per_node_log_size": "32", 2023-12-15T22:56:37.493 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2023-12-15T22:56:37.493 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_thread_pool_size": "512", 2023-12-15T22:56:37.493 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_topic_persistency_max_retries": "0", 2023-12-15T22:56:37.493 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_topic_persistency_sleep_duration": "0", 2023-12-15T22:56:37.493 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_topic_persistency_time_to_live": "0", 2023-12-15T22:56:37.493 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_torrent_comment": "", 2023-12-15T22:56:37.493 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_torrent_createby": "", 2023-12-15T22:56:37.493 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_torrent_encoding": "", 2023-12-15T22:56:37.493 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_torrent_flag": "false", 2023-12-15T22:56:37.493 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_torrent_max_size": "5368709120", 2023-12-15T22:56:37.494 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_torrent_origin": "", 2023-12-15T22:56:37.494 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_torrent_sha_unit": "524288", 2023-12-15T22:56:37.494 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_torrent_tracker": "", 2023-12-15T22:56:37.494 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_trust_forwarded_https": "false", 2023-12-15T22:56:37.494 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_usage_log_flush_threshold": "1024", 2023-12-15T22:56:37.494 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_usage_log_tick_interval": "30", 2023-12-15T22:56:37.494 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_usage_max_shards": "32", 2023-12-15T22:56:37.494 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_usage_max_user_shards": "1", 2023-12-15T22:56:37.494 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_use_opa_authz": "false", 2023-12-15T22:56:37.494 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_user_counters_cache": "false", 2023-12-15T22:56:37.494 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_user_counters_cache_size": "10000", 2023-12-15T22:56:37.494 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_user_default_quota_max_objects": "-1", 2023-12-15T22:56:37.494 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_user_default_quota_max_size": "-1", 2023-12-15T22:56:37.494 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_user_max_buckets": "1000", 2023-12-15T22:56:37.495 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_user_policies_max_num": "100", 2023-12-15T22:56:37.495 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2023-12-15T22:56:37.495 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_user_quota_sync_idle_users": "false", 2023-12-15T22:56:37.495 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_user_quota_sync_interval": "86400", 2023-12-15T22:56:37.495 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_user_quota_sync_wait_time": "86400", 2023-12-15T22:56:37.495 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_user_unique_email": "true", 2023-12-15T22:56:37.495 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_verify_ssl": "true", 2023-12-15T22:56:37.495 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_website_routing_rules_max_num": "50", 2023-12-15T22:56:37.495 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_zone": "", 2023-12-15T22:56:37.495 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_zone_id": "", 2023-12-15T22:56:37.495 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_zone_root_pool": ".rgw.root", 2023-12-15T22:56:37.495 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_zonegroup": "", 2023-12-15T22:56:37.495 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_zonegroup_id": "", 2023-12-15T22:56:37.495 INFO:teuthology.orchestra.run.smithi071.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2023-12-15T22:56:37.495 INFO:teuthology.orchestra.run.smithi071.stdout: "rgwlc_auto_session_clear": "true", 2023-12-15T22:56:37.496 INFO:teuthology.orchestra.run.smithi071.stdout: "rgwlc_skip_bucket_step": "false", 2023-12-15T22:56:37.496 INFO:teuthology.orchestra.run.smithi071.stdout: "rocksdb_block_size": "4096", 2023-12-15T22:56:37.496 INFO:teuthology.orchestra.run.smithi071.stdout: "rocksdb_bloom_bits_per_key": "20", 2023-12-15T22:56:37.496 INFO:teuthology.orchestra.run.smithi071.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2023-12-15T22:56:37.496 INFO:teuthology.orchestra.run.smithi071.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2023-12-15T22:56:37.496 INFO:teuthology.orchestra.run.smithi071.stdout: "rocksdb_cache_row_ratio": "0.000000", 2023-12-15T22:56:37.496 INFO:teuthology.orchestra.run.smithi071.stdout: "rocksdb_cache_shard_bits": "4", 2023-12-15T22:56:37.496 INFO:teuthology.orchestra.run.smithi071.stdout: "rocksdb_cache_size": "536870912", 2023-12-15T22:56:37.496 INFO:teuthology.orchestra.run.smithi071.stdout: "rocksdb_cache_type": "binned_lru", 2023-12-15T22:56:37.496 INFO:teuthology.orchestra.run.smithi071.stdout: "rocksdb_cf_compact_on_deletion": "true", 2023-12-15T22:56:37.496 INFO:teuthology.orchestra.run.smithi071.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2023-12-15T22:56:37.496 INFO:teuthology.orchestra.run.smithi071.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2023-12-15T22:56:37.496 INFO:teuthology.orchestra.run.smithi071.stdout: "rocksdb_collect_compaction_stats": "false", 2023-12-15T22:56:37.496 INFO:teuthology.orchestra.run.smithi071.stdout: "rocksdb_collect_extended_stats": "false", 2023-12-15T22:56:37.497 INFO:teuthology.orchestra.run.smithi071.stdout: "rocksdb_collect_memory_stats": "false", 2023-12-15T22:56:37.497 INFO:teuthology.orchestra.run.smithi071.stdout: "rocksdb_delete_range_threshold": "1048576", 2023-12-15T22:56:37.497 INFO:teuthology.orchestra.run.smithi071.stdout: "rocksdb_index_type": "binary_search", 2023-12-15T22:56:37.497 INFO:teuthology.orchestra.run.smithi071.stdout: "rocksdb_log_to_ceph_log": "true", 2023-12-15T22:56:37.497 INFO:teuthology.orchestra.run.smithi071.stdout: "rocksdb_metadata_block_size": "4096", 2023-12-15T22:56:37.497 INFO:teuthology.orchestra.run.smithi071.stdout: "rocksdb_partition_filters": "false", 2023-12-15T22:56:37.497 INFO:teuthology.orchestra.run.smithi071.stdout: "rocksdb_perf": "false", 2023-12-15T22:56:37.497 INFO:teuthology.orchestra.run.smithi071.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2023-12-15T22:56:37.497 INFO:teuthology.orchestra.run.smithi071.stdout: "rotating_keys_bootstrap_timeout": "30", 2023-12-15T22:56:37.497 INFO:teuthology.orchestra.run.smithi071.stdout: "rotating_keys_renewal_timeout": "10", 2023-12-15T22:56:37.497 INFO:teuthology.orchestra.run.smithi071.stdout: "run_dir": "/var/run/ceph", 2023-12-15T22:56:37.497 INFO:teuthology.orchestra.run.smithi071.stdout: "seastore_block_create": "true", 2023-12-15T22:56:37.497 INFO:teuthology.orchestra.run.smithi071.stdout: "seastore_cache_lru_size": "67108864", 2023-12-15T22:56:37.497 INFO:teuthology.orchestra.run.smithi071.stdout: "seastore_cbjournal_size": "5368709120", 2023-12-15T22:56:37.497 INFO:teuthology.orchestra.run.smithi071.stdout: "seastore_data_delta_based_overwrite": "0", 2023-12-15T22:56:37.498 INFO:teuthology.orchestra.run.smithi071.stdout: "seastore_default_max_object_size": "16777216", 2023-12-15T22:56:37.498 INFO:teuthology.orchestra.run.smithi071.stdout: "seastore_default_object_metadata_reservation": "16777216", 2023-12-15T22:56:37.498 INFO:teuthology.orchestra.run.smithi071.stdout: "seastore_device_size": "53687091200", 2023-12-15T22:56:37.498 INFO:teuthology.orchestra.run.smithi071.stdout: "seastore_journal_batch_capacity": "16", 2023-12-15T22:56:37.498 INFO:teuthology.orchestra.run.smithi071.stdout: "seastore_journal_batch_flush_size": "16777216", 2023-12-15T22:56:37.498 INFO:teuthology.orchestra.run.smithi071.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2023-12-15T22:56:37.498 INFO:teuthology.orchestra.run.smithi071.stdout: "seastore_journal_iodepth_limit": "5", 2023-12-15T22:56:37.498 INFO:teuthology.orchestra.run.smithi071.stdout: "seastore_main_device_type": "SSD", 2023-12-15T22:56:37.498 INFO:teuthology.orchestra.run.smithi071.stdout: "seastore_max_concurrent_transactions": "8", 2023-12-15T22:56:37.498 INFO:teuthology.orchestra.run.smithi071.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2023-12-15T22:56:37.498 INFO:teuthology.orchestra.run.smithi071.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2023-12-15T22:56:37.498 INFO:teuthology.orchestra.run.smithi071.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2023-12-15T22:56:37.498 INFO:teuthology.orchestra.run.smithi071.stdout: "seastore_obj_data_write_amplification": "1.250000", 2023-12-15T22:56:37.498 INFO:teuthology.orchestra.run.smithi071.stdout: "seastore_segment_size": "67108864", 2023-12-15T22:56:37.499 INFO:teuthology.orchestra.run.smithi071.stdout: "set_keepcaps": "false", 2023-12-15T22:56:37.499 INFO:teuthology.orchestra.run.smithi071.stdout: "setgroup": "", 2023-12-15T22:56:37.499 INFO:teuthology.orchestra.run.smithi071.stdout: "setuser": "", 2023-12-15T22:56:37.499 INFO:teuthology.orchestra.run.smithi071.stdout: "setuser_match_path": "", 2023-12-15T22:56:37.499 INFO:teuthology.orchestra.run.smithi071.stdout: "target_max_misplaced_ratio": "0.050000", 2023-12-15T22:56:37.499 INFO:teuthology.orchestra.run.smithi071.stdout: "thp": "false", 2023-12-15T22:56:37.499 INFO:teuthology.orchestra.run.smithi071.stdout: "threadpool_default_timeout": "60", 2023-12-15T22:56:37.499 INFO:teuthology.orchestra.run.smithi071.stdout: "threadpool_empty_queue_max_wait": "2", 2023-12-15T22:56:37.499 INFO:teuthology.orchestra.run.smithi071.stdout: "throttler_perf_counter": "true" 2023-12-15T22:56:37.499 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-12-15T22:56:37.501 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2023-12-15T22:56:37.605 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:37.602+0000 7fa7a8460640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:37.608 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:37.607+0000 7fa7a8460640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:37.620 INFO:teuthology.orchestra.run.smithi071.stdout:{} 2023-12-15T22:56:37.621 INFO:teuthology.orchestra.run.smithi071.stderr:mon_osd_down_out_interval = '' (not observed, change may require restart) 2023-12-15T22:56:37.631 DEBUG:teuthology.orchestra.run.smithi071:> type ceph-objectstore-tool 2023-12-15T22:56:37.644 INFO:teuthology.orchestra.run.smithi071.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2023-12-15T22:56:37.644 DEBUG:teuthology.orchestra.run.smithi072:> type ceph-objectstore-tool 2023-12-15T22:56:37.662 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2023-12-15T22:56:37.662 INFO:teuthology.run_tasks:Running task rados... 2023-12-15T22:56:37.673 INFO:tasks.rados:Beginning rados... 2023-12-15T22:56:37.674 DEBUG:teuthology.run_tasks:Unwinding manager rados 2023-12-15T22:56:37.702 INFO:tasks.rados:joining rados 2023-12-15T22:56:37.702 INFO:tasks.thrashosds.thrasher:starting do_thrash 2023-12-15T22:56:37.702 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T22:56:37.702 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T22:56:37.702 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-12-15T22:56:37.702 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-12-15T22:56:37.703 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2023-12-15T22:56:37.703 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:56:37.703 INFO:tasks.rados:clients are ['client.0', 'client.1'] 2023-12-15T22:56:37.703 INFO:tasks.rados:starting run 0 out of 1 2023-12-15T22:56:37.703 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2023-12-15T22:56:37.704 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2023-12-15T22:56:37.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:37.809+0000 7f470f19e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:37.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:37.813+0000 7f470f19e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:37.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:37.809+0000 7ff1eab5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:37.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:37.817+0000 7ff1eab5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:37.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:37.809+0000 7f01e7a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:37.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:37.821+0000 7f01e7a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:37.837 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:56:37.837 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:56:37.837 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:56:37.837 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:56:37.837 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:56:37.837 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:56:37.837 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:56:37.837 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:56:37.838 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:56:37.838 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:56:37.838 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:56:37.838 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:56:37.840 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:56:37.841 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:56:37.841 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-12-15T22:56:37.951 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:37.945+0000 7f762c91e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:37.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:37.953+0000 7f762c91e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:37.966 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:56:37.966 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:56:37.966 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:56:37.966 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:56:37.966 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:56:37.966 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:56:37.966 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:56:37.966 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:56:37.967 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:56:37.967 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:56:37.967 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:56:37.967 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:56:37.970 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:56:37.970 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:56:37.970 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-12-15T22:56:38.078 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:38.072+0000 7ff44e22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:38.081 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:38.080+0000 7ff44e22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:38.094 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:56:38.094 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:56:38.094 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:56:38.094 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:56:38.094 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:56:38.094 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:56:38.094 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:56:38.094 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:56:38.095 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:56:38.095 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:56:38.095 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:56:38.095 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:56:38.097 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:56:38.097 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:56:38.097 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-12-15T22:56:38.202 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:38.197+0000 7f00264ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:38.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:38.204+0000 7f00264ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:38.218 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:56:38.218 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:56:38.219 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:56:38.219 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:56:38.219 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:56:38.219 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:56:38.219 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:56:38.219 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:56:38.220 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:56:38.220 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:56:38.220 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:56:38.220 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:56:38.221 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:56:38.221 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:56:38.359 INFO:teuthology.orchestra.run.smithi071.stderr:pool 'unique_pool_0' created 2023-12-15T22:56:38.373 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2023-12-15T22:56:38.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:38.491+0000 7f4caa62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:38.504 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:38.503+0000 7f4caa62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:39.363 INFO:teuthology.orchestra.run.smithi071.stderr:enabled application 'rados' on pool 'unique_pool_0' 2023-12-15T22:56:39.896 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:56:39.999 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:39.998+0000 7f85206ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:40.002 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:40.002+0000 7f85206ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:40.379 DEBUG:teuthology.orchestra.run.smithi071:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0 2023-12-15T22:56:40.410 INFO:tasks.rados.rados.0.smithi071.stdout:adding op weight read -> 100 2023-12-15T22:56:40.411 INFO:tasks.rados.rados.0.smithi071.stdout:adding op weight write -> 50 2023-12-15T22:56:40.411 INFO:tasks.rados.rados.0.smithi071.stdout:adding op weight delete -> 50 2023-12-15T22:56:40.411 INFO:tasks.rados.rados.0.smithi071.stdout:adding op weight snap_create -> 50 2023-12-15T22:56:40.411 INFO:tasks.rados.rados.0.smithi071.stdout:adding op weight snap_remove -> 50 2023-12-15T22:56:40.411 INFO:tasks.rados.rados.0.smithi071.stdout:adding op weight write_excl -> 50 2023-12-15T22:56:40.411 INFO:tasks.rados.rados.0.smithi071.stdout:ceph version 19.0.0-201-gcd750a7f (cd750a7fc894b5dea3d055d595c0da884f63f94f) reef (dev) 2023-12-15T22:56:40.411 INFO:tasks.rados.rados.0.smithi071.stdout:Configuration: 2023-12-15T22:56:40.411 INFO:tasks.rados.rados.0.smithi071.stdout: Number of operations: 4000 2023-12-15T22:56:40.411 INFO:tasks.rados.rados.0.smithi071.stdout: Number of objects: 50 2023-12-15T22:56:40.411 INFO:tasks.rados.rados.0.smithi071.stdout: Max in flight operations: 16 2023-12-15T22:56:40.411 INFO:tasks.rados.rados.0.smithi071.stdout: Object size (in bytes): 4000000 2023-12-15T22:56:40.411 INFO:tasks.rados.rados.0.smithi071.stdout: Write stride min: 400000 2023-12-15T22:56:40.411 INFO:tasks.rados.rados.0.smithi071.stdout: Write stride max: 800000 2023-12-15T22:56:41.371 INFO:tasks.rados.rados.0.smithi071.stdout:1: write initial oid 1 2023-12-15T22:56:41.371 INFO:tasks.rados.rados.0.smithi071.stdout:1: seq_num 0 ranges {416807=475249,1542129=474678} 2023-12-15T22:56:41.376 INFO:tasks.rados.rados.0.smithi071.stdout:1: writing smithi07183189-1 from 416807 to 892056 tid 1 2023-12-15T22:56:41.383 INFO:tasks.rados.rados.0.smithi071.stdout:1: writing smithi07183189-1 from 1542129 to 2016807 tid 2 2023-12-15T22:56:41.383 INFO:tasks.rados.rados.0.smithi071.stdout:2: write initial oid 2 2023-12-15T22:56:41.383 INFO:tasks.rados.rados.0.smithi071.stdout:2: seq_num 1 ranges {416807=475249,1542129=474678} 2023-12-15T22:56:41.388 INFO:tasks.rados.rados.0.smithi071.stdout:2: writing smithi07183189-2 from 416807 to 892056 tid 1 2023-12-15T22:56:41.395 INFO:tasks.rados.rados.0.smithi071.stdout:2: writing smithi07183189-2 from 1542129 to 2016807 tid 2 2023-12-15T22:56:41.395 INFO:tasks.rados.rados.0.smithi071.stdout:3: write initial oid 3 2023-12-15T22:56:41.395 INFO:tasks.rados.rados.0.smithi071.stdout:3: seq_num 2 ranges {433614=550498,1600611=433003} 2023-12-15T22:56:41.400 INFO:tasks.rados.rados.0.smithi071.stdout:3: writing smithi07183189-3 from 433614 to 984112 tid 1 2023-12-15T22:56:41.407 INFO:tasks.rados.rados.0.smithi071.stdout:3: writing smithi07183189-3 from 1600611 to 2033614 tid 2 2023-12-15T22:56:41.407 INFO:tasks.rados.rados.0.smithi071.stdout:4: write initial oid 4 2023-12-15T22:56:41.407 INFO:tasks.rados.rados.0.smithi071.stdout:4: seq_num 3 ranges {450421=625747,1659093=391328} 2023-12-15T22:56:41.413 INFO:tasks.rados.rados.0.smithi071.stdout:4: writing smithi07183189-4 from 450421 to 1076168 tid 1 2023-12-15T22:56:41.419 INFO:tasks.rados.rados.0.smithi071.stdout:4: writing smithi07183189-4 from 1659093 to 2050421 tid 2 2023-12-15T22:56:41.419 INFO:tasks.rados.rados.0.smithi071.stdout:5: write initial oid 5 2023-12-15T22:56:41.419 INFO:tasks.rados.rados.0.smithi071.stdout:5: seq_num 4 ranges {467228=700996,1717575=349653} 2023-12-15T22:56:41.425 INFO:tasks.rados.rados.0.smithi071.stdout:5: writing smithi07183189-5 from 467228 to 1168224 tid 1 2023-12-15T22:56:41.431 INFO:tasks.rados.rados.0.smithi071.stdout:5: writing smithi07183189-5 from 1717575 to 2067228 tid 2 2023-12-15T22:56:41.431 INFO:tasks.rados.rados.0.smithi071.stdout:6: write initial oid 6 2023-12-15T22:56:41.431 INFO:tasks.rados.rados.0.smithi071.stdout:6: seq_num 5 ranges {484035=776245,2059704=24331} 2023-12-15T22:56:41.438 INFO:tasks.rados.rados.0.smithi071.stdout:6: writing smithi07183189-6 from 484035 to 1260280 tid 1 2023-12-15T22:56:41.443 INFO:tasks.rados.rados.0.smithi071.stdout:6: writing smithi07183189-6 from 2059704 to 2084035 tid 2 2023-12-15T22:56:41.443 INFO:tasks.rados.rados.0.smithi071.stdout:7: write initial oid 7 2023-12-15T22:56:41.443 INFO:tasks.rados.rados.0.smithi071.stdout:7: seq_num 6 ranges {500842=451494,1718186=382656} 2023-12-15T22:56:41.449 INFO:tasks.rados.rados.0.smithi071.stdout:7: writing smithi07183189-7 from 500842 to 952336 tid 1 2023-12-15T22:56:41.456 INFO:tasks.rados.rados.0.smithi071.stdout:7: writing smithi07183189-7 from 1718186 to 2100842 tid 2 2023-12-15T22:56:41.456 INFO:tasks.rados.rados.0.smithi071.stdout:8: write initial oid 8 2023-12-15T22:56:41.456 INFO:tasks.rados.rados.0.smithi071.stdout:8: seq_num 7 ranges {517649=526743,1776668=340981} 2023-12-15T22:56:41.462 INFO:tasks.rados.rados.0.smithi071.stdout:8: writing smithi07183189-8 from 517649 to 1044392 tid 1 2023-12-15T22:56:41.468 INFO:tasks.rados.rados.0.smithi071.stdout:8: writing smithi07183189-8 from 1776668 to 2117649 tid 2 2023-12-15T22:56:41.469 INFO:tasks.rados.rados.0.smithi071.stdout:9: write initial oid 9 2023-12-15T22:56:41.469 INFO:tasks.rados.rados.0.smithi071.stdout:9: seq_num 8 ranges {534456=718345,1951503=182953} 2023-12-15T22:56:41.476 INFO:tasks.rados.rados.0.smithi071.stdout:9: writing smithi07183189-9 from 534456 to 1252801 tid 1 2023-12-15T22:56:41.481 INFO:tasks.rados.rados.0.smithi071.stdout:9: writing smithi07183189-9 from 1951503 to 2134456 tid 2 2023-12-15T22:56:41.481 INFO:tasks.rados.rados.0.smithi071.stdout:10: write initial oid 10 2023-12-15T22:56:41.481 INFO:tasks.rados.rados.0.smithi071.stdout:10: seq_num 9 ranges {551263=793594,1893632=257631} 2023-12-15T22:56:41.489 INFO:tasks.rados.rados.0.smithi071.stdout:10: writing smithi07183189-10 from 551263 to 1344857 tid 1 2023-12-15T22:56:41.494 INFO:tasks.rados.rados.0.smithi071.stdout:10: writing smithi07183189-10 from 1893632 to 2151263 tid 2 2023-12-15T22:56:41.494 INFO:tasks.rados.rados.0.smithi071.stdout:11: write initial oid 11 2023-12-15T22:56:41.494 INFO:tasks.rados.rados.0.smithi071.stdout:11: seq_num 10 ranges {568070=468843,1552114=615956} 2023-12-15T22:56:41.500 INFO:tasks.rados.rados.0.smithi071.stdout:11: writing smithi07183189-11 from 568070 to 1036913 tid 1 2023-12-15T22:56:41.507 INFO:tasks.rados.rados.0.smithi071.stdout:11: writing smithi07183189-11 from 1552114 to 2168070 tid 2 2023-12-15T22:56:41.507 INFO:tasks.rados.rados.0.smithi071.stdout:12: write initial oid 12 2023-12-15T22:56:41.507 INFO:tasks.rados.rados.0.smithi071.stdout:12: seq_num 11 ranges {584877=544092,1610596=562003} 2023-12-15T22:56:41.513 INFO:tasks.rados.rados.0.smithi071.stdout:12: writing smithi07183189-12 from 584877 to 1128969 tid 1 2023-12-15T22:56:41.519 INFO:tasks.rados.rados.0.smithi071.stdout:12: writing smithi07183189-12 from 1610596 to 2172599 tid 2 2023-12-15T22:56:41.519 INFO:tasks.rados.rados.0.smithi071.stdout:13: write initial oid 13 2023-12-15T22:56:41.519 INFO:tasks.rados.rados.0.smithi071.stdout:13: seq_num 12 ranges {601684=619341,1669078=532606} 2023-12-15T22:56:41.526 INFO:tasks.rados.rados.0.smithi071.stdout:13: writing smithi07183189-13 from 601684 to 1221025 tid 1 2023-12-15T22:56:41.533 INFO:tasks.rados.rados.0.smithi071.stdout:13: writing smithi07183189-13 from 1669078 to 2201684 tid 2 2023-12-15T22:56:41.533 INFO:tasks.rados.rados.0.smithi071.stdout:14: write initial oid 14 2023-12-15T22:56:41.533 INFO:tasks.rados.rados.0.smithi071.stdout:14: seq_num 13 ranges {618491=694590,2011207=207284} 2023-12-15T22:56:41.540 INFO:tasks.rados.rados.0.smithi071.stdout:14: writing smithi07183189-14 from 618491 to 1313081 tid 1 2023-12-15T22:56:41.546 INFO:tasks.rados.rados.0.smithi071.stdout:14: writing smithi07183189-14 from 2011207 to 2218491 tid 2 2023-12-15T22:56:41.546 INFO:tasks.rados.rados.0.smithi071.stdout:15: write initial oid 15 2023-12-15T22:56:41.546 INFO:tasks.rados.rados.0.smithi071.stdout:15: seq_num 14 ranges {635298=769839,2069689=165609} 2023-12-15T22:56:41.554 INFO:tasks.rados.rados.0.smithi071.stdout:15: writing smithi07183189-15 from 635298 to 1405137 tid 1 2023-12-15T22:56:41.559 INFO:tasks.rados.rados.0.smithi071.stdout:15: writing smithi07183189-15 from 2069689 to 2235298 tid 2 2023-12-15T22:56:41.559 INFO:tasks.rados.rados.0.smithi071.stdout:16: write initial oid 16 2023-12-15T22:56:41.559 INFO:tasks.rados.rados.0.smithi071.stdout:16: seq_num 15 ranges {652105=445088,1728171=452988} 2023-12-15T22:56:41.565 INFO:tasks.rados.rados.0.smithi071.stdout:16: writing smithi07183189-16 from 652105 to 1097193 tid 1 2023-12-15T22:56:41.572 INFO:tasks.rados.rados.0.smithi071.stdout:16: writing smithi07183189-16 from 1728171 to 2181159 tid 2 2023-12-15T22:56:41.572 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2023-12-15T22:56:41.572 INFO:tasks.rados.rados.0.smithi071.stdout:1: finishing write tid 1 to smithi07183189-1 2023-12-15T22:56:41.572 INFO:tasks.rados.rados.0.smithi071.stdout:2: finishing write tid 1 to smithi07183189-2 2023-12-15T22:56:41.572 INFO:tasks.rados.rados.0.smithi071.stdout:1: finishing write tid 2 to smithi07183189-1 2023-12-15T22:56:41.572 INFO:tasks.rados.rados.0.smithi071.stdout:5: finishing write tid 1 to smithi07183189-5 2023-12-15T22:56:41.572 INFO:tasks.rados.rados.0.smithi071.stdout:2: finishing write tid 2 to smithi07183189-2 2023-12-15T22:56:41.572 INFO:tasks.rados.rados.0.smithi071.stdout:2: finishing write tid 3 to smithi07183189-2 2023-12-15T22:56:41.572 INFO:tasks.rados.rados.0.smithi071.stdout:2: finishing write tid 4 to smithi07183189-2 2023-12-15T22:56:41.573 INFO:tasks.rados.rados.0.smithi071.stdout:2: oid 2 updating version 0 to 1 2023-12-15T22:56:41.573 INFO:tasks.rados.rados.0.smithi071.stdout:2: oid 2 updating version 1 to 3 2023-12-15T22:56:41.573 INFO:tasks.rados.rados.0.smithi071.stdout:2: oid 2 version 3 is already newer than 3 2023-12-15T22:56:41.573 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2023-12-15T22:56:41.573 INFO:tasks.rados.rados.0.smithi071.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2023-12-15T22:56:41.573 INFO:tasks.rados.rados.0.smithi071.stdout:1: finishing write tid 3 to smithi07183189-1 2023-12-15T22:56:41.573 INFO:tasks.rados.rados.0.smithi071.stdout:1: finishing write tid 4 to smithi07183189-1 2023-12-15T22:56:41.573 INFO:tasks.rados.rados.0.smithi071.stdout:1: oid 1 updating version 0 to 2 2023-12-15T22:56:41.573 INFO:tasks.rados.rados.0.smithi071.stdout:1: oid 1 version 2 is already newer than 1 2023-12-15T22:56:41.573 INFO:tasks.rados.rados.0.smithi071.stdout:1: oid 1 updating version 2 to 3 2023-12-15T22:56:41.573 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2023-12-15T22:56:41.573 INFO:tasks.rados.rados.0.smithi071.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2023-12-15T22:56:41.573 INFO:tasks.rados.rados.0.smithi071.stdout:9: finishing write tid 1 to smithi07183189-9 2023-12-15T22:56:41.573 INFO:tasks.rados.rados.0.smithi071.stdout:3: finishing write tid 1 to smithi07183189-3 2023-12-15T22:56:41.574 INFO:tasks.rados.rados.0.smithi071.stdout:5: finishing write tid 2 to smithi07183189-5 2023-12-15T22:56:41.574 INFO:tasks.rados.rados.0.smithi071.stdout:1: done (15 left) 2023-12-15T22:56:41.574 INFO:tasks.rados.rados.0.smithi071.stdout:2: done (14 left) 2023-12-15T22:56:41.574 INFO:tasks.rados.rados.0.smithi071.stdout:17: write initial oid 17 2023-12-15T22:56:41.574 INFO:tasks.rados.rados.0.smithi071.stdout:17: seq_num 16 ranges {668912=636690,1903006=365906} 2023-12-15T22:56:41.579 INFO:tasks.rados.rados.0.smithi071.stdout:17: writing smithi07183189-17 from 668912 to 1305602 tid 1 2023-12-15T22:56:41.585 INFO:tasks.rados.rados.0.smithi071.stdout:17: writing smithi07183189-17 from 1903006 to 2268912 tid 2 2023-12-15T22:56:41.585 INFO:tasks.rados.rados.0.smithi071.stdout:18: write initial oid 18 2023-12-15T22:56:41.585 INFO:tasks.rados.rados.0.smithi071.stdout:18: seq_num 17 ranges {685719=711939,1845135=440584} 2023-12-15T22:56:41.593 INFO:tasks.rados.rados.0.smithi071.stdout:18: writing smithi07183189-18 from 685719 to 1397658 tid 1 2023-12-15T22:56:41.599 INFO:tasks.rados.rados.0.smithi071.stdout:18: writing smithi07183189-18 from 1845135 to 2285719 tid 2 2023-12-15T22:56:41.599 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2023-12-15T22:56:41.599 INFO:tasks.rados.rados.0.smithi071.stdout:4: finishing write tid 1 to smithi07183189-4 2023-12-15T22:56:41.599 INFO:tasks.rados.rados.0.smithi071.stdout:9: finishing write tid 2 to smithi07183189-9 2023-12-15T22:56:41.599 INFO:tasks.rados.rados.0.smithi071.stdout:8: finishing write tid 1 to smithi07183189-8 2023-12-15T22:56:41.599 INFO:tasks.rados.rados.0.smithi071.stdout:9: finishing write tid 3 to smithi07183189-9 2023-12-15T22:56:41.599 INFO:tasks.rados.rados.0.smithi071.stdout:12: finishing write tid 1 to smithi07183189-12 2023-12-15T22:56:41.600 INFO:tasks.rados.rados.0.smithi071.stdout:9: finishing write tid 4 to smithi07183189-9 2023-12-15T22:56:41.600 INFO:tasks.rados.rados.0.smithi071.stdout:9: oid 9 updating version 0 to 3 2023-12-15T22:56:41.600 INFO:tasks.rados.rados.0.smithi071.stdout:9: oid 9 version 3 is already newer than 3 2023-12-15T22:56:41.600 INFO:tasks.rados.rados.0.smithi071.stdout:9: oid 9 version 3 is already newer than 1 2023-12-15T22:56:41.600 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 3 (ObjNum 8 snap 0 seq_num 8) dirty exists 2023-12-15T22:56:41.600 INFO:tasks.rados.rados.0.smithi071.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2023-12-15T22:56:41.600 INFO:tasks.rados.rados.0.smithi071.stdout:3: finishing write tid 2 to smithi07183189-3 2023-12-15T22:56:41.600 INFO:tasks.rados.rados.0.smithi071.stdout:9: done (15 left) 2023-12-15T22:56:41.600 INFO:tasks.rados.rados.0.smithi071.stdout:19: write initial oid 19 2023-12-15T22:56:41.600 INFO:tasks.rados.rados.0.smithi071.stdout:19: seq_num 18 ranges {702526=787188,1903617=398909} 2023-12-15T22:56:41.607 INFO:tasks.rados.rados.0.smithi071.stdout:19: writing smithi07183189-19 from 702526 to 1489714 tid 1 2023-12-15T22:56:41.613 INFO:tasks.rados.rados.0.smithi071.stdout:19: writing smithi07183189-19 from 1903617 to 2302526 tid 2 2023-12-15T22:56:41.613 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2023-12-15T22:56:41.613 INFO:tasks.rados.rados.0.smithi071.stdout:10: finishing write tid 1 to smithi07183189-10 2023-12-15T22:56:41.614 INFO:tasks.rados.rados.0.smithi071.stdout:5: finishing write tid 3 to smithi07183189-5 2023-12-15T22:56:41.614 INFO:tasks.rados.rados.0.smithi071.stdout:5: finishing write tid 4 to smithi07183189-5 2023-12-15T22:56:41.614 INFO:tasks.rados.rados.0.smithi071.stdout:5: oid 5 updating version 0 to 3 2023-12-15T22:56:41.614 INFO:tasks.rados.rados.0.smithi071.stdout:5: oid 5 version 3 is already newer than 1 2023-12-15T22:56:41.614 INFO:tasks.rados.rados.0.smithi071.stdout:5: oid 5 version 3 is already newer than 2 2023-12-15T22:56:41.614 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2023-12-15T22:56:41.614 INFO:tasks.rados.rados.0.smithi071.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2023-12-15T22:56:41.614 INFO:tasks.rados.rados.0.smithi071.stdout:5: done (15 left) 2023-12-15T22:56:41.614 INFO:tasks.rados.rados.0.smithi071.stdout:20: write initial oid 20 2023-12-15T22:56:41.615 INFO:tasks.rados.rados.0.smithi071.stdout:20: seq_num 19 ranges {719333=462437,1962099=357234} 2023-12-15T22:56:41.619 INFO:tasks.rados.rados.0.smithi071.stdout:20: writing smithi07183189-20 from 719333 to 1181770 tid 1 2023-12-15T22:56:41.626 INFO:tasks.rados.rados.0.smithi071.stdout:20: writing smithi07183189-20 from 1962099 to 2319333 tid 2 2023-12-15T22:56:41.626 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2023-12-15T22:56:41.626 INFO:tasks.rados.rados.0.smithi071.stdout:4: finishing write tid 2 to smithi07183189-4 2023-12-15T22:56:41.626 INFO:tasks.rados.rados.0.smithi071.stdout:4: finishing write tid 3 to smithi07183189-4 2023-12-15T22:56:41.626 INFO:tasks.rados.rados.0.smithi071.stdout:8: finishing write tid 2 to smithi07183189-8 2023-12-15T22:56:41.631 INFO:tasks.rados.rados.0.smithi071.stdout:4: finishing write tid 4 to smithi07183189-4 2023-12-15T22:56:41.631 INFO:tasks.rados.rados.0.smithi071.stdout:4: oid 4 updating version 0 to 1 2023-12-15T22:56:41.631 INFO:tasks.rados.rados.0.smithi071.stdout:4: oid 4 updating version 1 to 3 2023-12-15T22:56:41.631 INFO:tasks.rados.rados.0.smithi071.stdout:4: oid 4 version 3 is already newer than 3 2023-12-15T22:56:41.631 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2023-12-15T22:56:41.632 INFO:tasks.rados.rados.0.smithi071.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2023-12-15T22:56:41.632 INFO:tasks.rados.rados.0.smithi071.stdout:4: done (15 left) 2023-12-15T22:56:41.632 INFO:tasks.rados.rados.0.smithi071.stdout:21: write initial oid 21 2023-12-15T22:56:41.632 INFO:tasks.rados.rados.0.smithi071.stdout:21: seq_num 20 ranges {736140=537686,2020581=315559} 2023-12-15T22:56:41.638 INFO:tasks.rados.rados.0.smithi071.stdout:21: writing smithi07183189-21 from 736140 to 1273826 tid 1 2023-12-15T22:56:41.644 INFO:tasks.rados.rados.0.smithi071.stdout:21: writing smithi07183189-21 from 2020581 to 2336140 tid 2 2023-12-15T22:56:41.644 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2023-12-15T22:56:41.644 INFO:tasks.rados.rados.0.smithi071.stdout:3: finishing write tid 3 to smithi07183189-3 2023-12-15T22:56:41.644 INFO:tasks.rados.rados.0.smithi071.stdout:12: finishing write tid 2 to smithi07183189-12 2023-12-15T22:56:41.645 INFO:tasks.rados.rados.0.smithi071.stdout:3: finishing write tid 4 to smithi07183189-3 2023-12-15T22:56:41.645 INFO:tasks.rados.rados.0.smithi071.stdout:3: oid 3 updating version 0 to 1 2023-12-15T22:56:41.645 INFO:tasks.rados.rados.0.smithi071.stdout:3: oid 3 updating version 1 to 2 2023-12-15T22:56:41.645 INFO:tasks.rados.rados.0.smithi071.stdout:3: oid 3 updating version 2 to 3 2023-12-15T22:56:41.645 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2023-12-15T22:56:41.645 INFO:tasks.rados.rados.0.smithi071.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2023-12-15T22:56:41.645 INFO:tasks.rados.rados.0.smithi071.stdout:3: done (15 left) 2023-12-15T22:56:41.645 INFO:tasks.rados.rados.0.smithi071.stdout:22: write initial oid 22 2023-12-15T22:56:41.645 INFO:tasks.rados.rados.0.smithi071.stdout:22: seq_num 21 ranges {752947=612935,1962710=390237} 2023-12-15T22:56:41.654 INFO:tasks.rados.rados.0.smithi071.stdout:22: writing smithi07183189-22 from 752947 to 1365882 tid 1 2023-12-15T22:56:41.661 INFO:tasks.rados.rados.0.smithi071.stdout:22: writing smithi07183189-22 from 1962710 to 2352947 tid 2 2023-12-15T22:56:41.661 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2023-12-15T22:56:41.661 INFO:tasks.rados.rados.0.smithi071.stdout:12: finishing write tid 3 to smithi07183189-12 2023-12-15T22:56:41.661 INFO:tasks.rados.rados.0.smithi071.stdout:12: finishing write tid 4 to smithi07183189-12 2023-12-15T22:56:41.661 INFO:tasks.rados.rados.0.smithi071.stdout:12: oid 12 updating version 0 to 3 2023-12-15T22:56:41.662 INFO:tasks.rados.rados.0.smithi071.stdout:12: oid 12 version 3 is already newer than 3 2023-12-15T22:56:41.662 INFO:tasks.rados.rados.0.smithi071.stdout:12: oid 12 version 3 is already newer than 1 2023-12-15T22:56:41.662 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 12 v 3 (ObjNum 11 snap 0 seq_num 11) dirty exists 2023-12-15T22:56:41.662 INFO:tasks.rados.rados.0.smithi071.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2023-12-15T22:56:41.662 INFO:tasks.rados.rados.0.smithi071.stdout:12: done (15 left) 2023-12-15T22:56:41.662 INFO:tasks.rados.rados.0.smithi071.stdout:23: write initial oid 23 2023-12-15T22:56:41.662 INFO:tasks.rados.rados.0.smithi071.stdout:23: seq_num 22 ranges {769754=688184,2021192=348562} 2023-12-15T22:56:41.671 INFO:tasks.rados.rados.0.smithi071.stdout:23: writing smithi07183189-23 from 769754 to 1457938 tid 1 2023-12-15T22:56:41.678 INFO:tasks.rados.rados.0.smithi071.stdout:23: writing smithi07183189-23 from 2021192 to 2369754 tid 2 2023-12-15T22:56:41.678 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2023-12-15T22:56:41.691 INFO:tasks.rados.rados.0.smithi071.stdout:10: finishing write tid 2 to smithi07183189-10 2023-12-15T22:56:41.712 INFO:tasks.rados.rados.0.smithi071.stdout:7: finishing write tid 1 to smithi07183189-7 2023-12-15T22:56:41.718 INFO:tasks.rados.rados.0.smithi071.stdout:18: finishing write tid 1 to smithi07183189-18 2023-12-15T22:56:41.737 INFO:tasks.rados.rados.0.smithi071.stdout:8: finishing write tid 3 to smithi07183189-8 2023-12-15T22:56:41.738 INFO:tasks.rados.rados.0.smithi071.stdout:6: finishing write tid 1 to smithi07183189-6 2023-12-15T22:56:41.758 INFO:tasks.rados.rados.0.smithi071.stdout:8: finishing write tid 4 to smithi07183189-8 2023-12-15T22:56:41.758 INFO:tasks.rados.rados.0.smithi071.stdout:8: oid 8 updating version 0 to 2 2023-12-15T22:56:41.759 INFO:tasks.rados.rados.0.smithi071.stdout:8: oid 8 updating version 2 to 3 2023-12-15T22:56:41.759 INFO:tasks.rados.rados.0.smithi071.stdout:8: oid 8 version 3 is already newer than 1 2023-12-15T22:56:41.759 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 8 v 3 (ObjNum 7 snap 0 seq_num 7) dirty exists 2023-12-15T22:56:41.759 INFO:tasks.rados.rados.0.smithi071.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2023-12-15T22:56:41.759 INFO:tasks.rados.rados.0.smithi071.stdout:8: done (15 left) 2023-12-15T22:56:41.759 INFO:tasks.rados.rados.0.smithi071.stdout:24: write initial oid 24 2023-12-15T22:56:41.759 INFO:tasks.rados.rados.0.smithi071.stdout:24: seq_num 23 ranges {786561=479786,1796027=467664} 2023-12-15T22:56:41.765 INFO:tasks.rados.rados.0.smithi071.stdout:24: writing smithi07183189-24 from 786561 to 1266347 tid 1 2023-12-15T22:56:41.772 INFO:tasks.rados.rados.0.smithi071.stdout:24: writing smithi07183189-24 from 1796027 to 2263691 tid 2 2023-12-15T22:56:41.772 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2023-12-15T22:56:41.772 INFO:tasks.rados.rados.0.smithi071.stdout:19: finishing write tid 1 to smithi07183189-19 2023-12-15T22:56:41.772 INFO:tasks.rados.rados.0.smithi071.stdout:22: finishing write tid 1 to smithi07183189-22 2023-12-15T22:56:41.777 INFO:tasks.rados.rados.0.smithi071.stdout:10: finishing write tid 3 to smithi07183189-10 2023-12-15T22:56:41.779 INFO:tasks.rados.rados.0.smithi071.stdout:10: finishing write tid 4 to smithi07183189-10 2023-12-15T22:56:41.779 INFO:tasks.rados.rados.0.smithi071.stdout:10: oid 10 updating version 0 to 2 2023-12-15T22:56:41.779 INFO:tasks.rados.rados.0.smithi071.stdout:10: oid 10 version 2 is already newer than 1 2023-12-15T22:56:41.779 INFO:tasks.rados.rados.0.smithi071.stdout:10: oid 10 updating version 2 to 3 2023-12-15T22:56:41.779 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 3 (ObjNum 9 snap 0 seq_num 9) dirty exists 2023-12-15T22:56:41.779 INFO:tasks.rados.rados.0.smithi071.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2023-12-15T22:56:41.779 INFO:tasks.rados.rados.0.smithi071.stdout:10: done (15 left) 2023-12-15T22:56:41.779 INFO:tasks.rados.rados.0.smithi071.stdout:25: write initial oid 25 2023-12-15T22:56:41.779 INFO:tasks.rados.rados.0.smithi071.stdout:25: seq_num 24 ranges {403368=555035,1454509=727675} 2023-12-15T22:56:41.784 INFO:tasks.rados.rados.0.smithi071.stdout:25: writing smithi07183189-25 from 403368 to 958403 tid 1 2023-12-15T22:56:41.792 INFO:tasks.rados.rados.0.smithi071.stdout:25: writing smithi07183189-25 from 1454509 to 2182184 tid 2 2023-12-15T22:56:41.792 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2023-12-15T22:56:41.794 INFO:tasks.rados.rados.0.smithi071.stdout:6: finishing write tid 2 to smithi07183189-6 2023-12-15T22:56:41.802 INFO:tasks.rados.rados.0.smithi071.stdout:18: finishing write tid 2 to smithi07183189-18 2023-12-15T22:56:41.805 INFO:tasks.rados.rados.0.smithi071.stdout:23: finishing write tid 1 to smithi07183189-23 2023-12-15T22:56:41.805 INFO:tasks.rados.rados.0.smithi071.stdout:18: finishing write tid 3 to smithi07183189-18 2023-12-15T22:56:41.805 INFO:tasks.rados.rados.0.smithi071.stdout:18: finishing write tid 4 to smithi07183189-18 2023-12-15T22:56:41.806 INFO:tasks.rados.rados.0.smithi071.stdout:18: oid 18 updating version 0 to 6 2023-12-15T22:56:41.806 INFO:tasks.rados.rados.0.smithi071.stdout:18: oid 18 version 6 is already newer than 6 2023-12-15T22:56:41.806 INFO:tasks.rados.rados.0.smithi071.stdout:18: oid 18 version 6 is already newer than 4 2023-12-15T22:56:41.806 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 6 (ObjNum 17 snap 0 seq_num 17) dirty exists 2023-12-15T22:56:41.806 INFO:tasks.rados.rados.0.smithi071.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2023-12-15T22:56:41.806 INFO:tasks.rados.rados.0.smithi071.stdout:18: done (15 left) 2023-12-15T22:56:41.806 INFO:tasks.rados.rados.0.smithi071.stdout:26: write initial oid 26 2023-12-15T22:56:41.806 INFO:tasks.rados.rados.0.smithi071.stdout:26: seq_num 25 ranges {420175=630284,1796638=471333} 2023-12-15T22:56:41.812 INFO:tasks.rados.rados.0.smithi071.stdout:26: writing smithi07183189-26 from 420175 to 1050459 tid 1 2023-12-15T22:56:41.820 INFO:tasks.rados.rados.0.smithi071.stdout:26: writing smithi07183189-26 from 1796638 to 2267971 tid 2 2023-12-15T22:56:41.820 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2023-12-15T22:56:41.820 INFO:tasks.rados.rados.0.smithi071.stdout:20: finishing write tid 1 to smithi07183189-20 2023-12-15T22:56:41.820 INFO:tasks.rados.rados.0.smithi071.stdout:7: finishing write tid 2 to smithi07183189-7 2023-12-15T22:56:41.820 INFO:tasks.rados.rados.0.smithi071.stdout:13: finishing write tid 1 to smithi07183189-13 2023-12-15T22:56:41.822 INFO:tasks.rados.rados.0.smithi071.stdout:24: finishing write tid 1 to smithi07183189-24 2023-12-15T22:56:41.839 INFO:tasks.rados.rados.0.smithi071.stdout:19: finishing write tid 2 to smithi07183189-19 2023-12-15T22:56:41.841 INFO:tasks.rados.rados.0.smithi071.stdout:22: finishing write tid 2 to smithi07183189-22 2023-12-15T22:56:41.850 INFO:tasks.rados.rados.0.smithi071.stdout:11: finishing write tid 1 to smithi07183189-11 2023-12-15T22:56:41.856 INFO:tasks.rados.rados.0.smithi071.stdout:23: finishing write tid 2 to smithi07183189-23 2023-12-15T22:56:41.858 INFO:tasks.rados.rados.0.smithi071.stdout:20: finishing write tid 2 to smithi07183189-20 2023-12-15T22:56:41.859 INFO:tasks.rados.rados.0.smithi071.stdout:6: finishing write tid 3 to smithi07183189-6 2023-12-15T22:56:41.860 INFO:tasks.rados.rados.0.smithi071.stdout:6: finishing write tid 4 to smithi07183189-6 2023-12-15T22:56:41.861 INFO:tasks.rados.rados.0.smithi071.stdout:6: oid 6 updating version 0 to 5 2023-12-15T22:56:41.861 INFO:tasks.rados.rados.0.smithi071.stdout:6: oid 6 version 5 is already newer than 4 2023-12-15T22:56:41.861 INFO:tasks.rados.rados.0.smithi071.stdout:6: oid 6 updating version 5 to 6 2023-12-15T22:56:41.861 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 6 v 6 (ObjNum 5 snap 0 seq_num 5) dirty exists 2023-12-15T22:56:41.861 INFO:tasks.rados.rados.0.smithi071.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2023-12-15T22:56:41.861 INFO:tasks.rados.rados.0.smithi071.stdout:6: done (15 left) 2023-12-15T22:56:41.861 INFO:tasks.rados.rados.0.smithi071.stdout:27: write initial oid 27 2023-12-15T22:56:41.861 INFO:tasks.rados.rados.0.smithi071.stdout:27: seq_num 26 ranges {436982=705533,1855120=581862} 2023-12-15T22:56:41.869 INFO:tasks.rados.rados.0.smithi071.stdout:27: writing smithi07183189-27 from 436982 to 1142515 tid 1 2023-12-15T22:56:41.878 INFO:tasks.rados.rados.0.smithi071.stdout:27: writing smithi07183189-27 from 1855120 to 2436982 tid 2 2023-12-15T22:56:41.878 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2023-12-15T22:56:41.878 INFO:tasks.rados.rados.0.smithi071.stdout:22: finishing write tid 3 to smithi07183189-22 2023-12-15T22:56:41.878 INFO:tasks.rados.rados.0.smithi071.stdout:22: finishing write tid 4 to smithi07183189-22 2023-12-15T22:56:41.878 INFO:tasks.rados.rados.0.smithi071.stdout:22: oid 22 updating version 0 to 3 2023-12-15T22:56:41.878 INFO:tasks.rados.rados.0.smithi071.stdout:22: oid 22 version 3 is already newer than 2 2023-12-15T22:56:41.878 INFO:tasks.rados.rados.0.smithi071.stdout:22: oid 22 version 3 is already newer than 1 2023-12-15T22:56:41.878 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 3 (ObjNum 21 snap 0 seq_num 21) dirty exists 2023-12-15T22:56:41.878 INFO:tasks.rados.rados.0.smithi071.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2023-12-15T22:56:41.878 INFO:tasks.rados.rados.0.smithi071.stdout:24: finishing write tid 2 to smithi07183189-24 2023-12-15T22:56:41.878 INFO:tasks.rados.rados.0.smithi071.stdout:22: done (15 left) 2023-12-15T22:56:41.878 INFO:tasks.rados.rados.0.smithi071.stdout:28: write initial oid 28 2023-12-15T22:56:41.878 INFO:tasks.rados.rados.0.smithi071.stdout:28: seq_num 27 ranges {453789=780782,1913602=475002} 2023-12-15T22:56:41.884 INFO:tasks.rados.rados.0.smithi071.stdout:28: writing smithi07183189-28 from 453789 to 1234571 tid 1 2023-12-15T22:56:41.892 INFO:tasks.rados.rados.0.smithi071.stdout:28: writing smithi07183189-28 from 1913602 to 2388604 tid 2 2023-12-15T22:56:41.892 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2023-12-15T22:56:41.892 INFO:tasks.rados.rados.0.smithi071.stdout:13: finishing write tid 2 to smithi07183189-13 2023-12-15T22:56:41.892 INFO:tasks.rados.rados.0.smithi071.stdout:19: finishing write tid 3 to smithi07183189-19 2023-12-15T22:56:41.892 INFO:tasks.rados.rados.0.smithi071.stdout:7: finishing write tid 3 to smithi07183189-7 2023-12-15T22:56:41.894 INFO:tasks.rados.rados.0.smithi071.stdout:7: finishing write tid 4 to smithi07183189-7 2023-12-15T22:56:41.894 INFO:tasks.rados.rados.0.smithi071.stdout:7: oid 7 updating version 0 to 4 2023-12-15T22:56:41.894 INFO:tasks.rados.rados.0.smithi071.stdout:7: oid 7 updating version 4 to 6 2023-12-15T22:56:41.894 INFO:tasks.rados.rados.0.smithi071.stdout:7: oid 7 version 6 is already newer than 5 2023-12-15T22:56:41.894 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 6 (ObjNum 6 snap 0 seq_num 6) dirty exists 2023-12-15T22:56:41.894 INFO:tasks.rados.rados.0.smithi071.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2023-12-15T22:56:41.894 INFO:tasks.rados.rados.0.smithi071.stdout:7: done (15 left) 2023-12-15T22:56:41.894 INFO:tasks.rados.rados.0.smithi071.stdout:29: write initial oid 29 2023-12-15T22:56:41.894 INFO:tasks.rados.rados.0.smithi071.stdout:29: seq_num 28 ranges {470596=456031,1572084=618660} 2023-12-15T22:56:41.900 INFO:tasks.rados.rados.0.smithi071.stdout:29: writing smithi07183189-29 from 470596 to 926627 tid 1 2023-12-15T22:56:41.909 INFO:tasks.rados.rados.0.smithi071.stdout:29: writing smithi07183189-29 from 1572084 to 2190744 tid 2 2023-12-15T22:56:41.909 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2023-12-15T22:56:41.909 INFO:tasks.rados.rados.0.smithi071.stdout:19: finishing write tid 4 to smithi07183189-19 2023-12-15T22:56:41.909 INFO:tasks.rados.rados.0.smithi071.stdout:19: oid 19 updating version 0 to 6 2023-12-15T22:56:41.910 INFO:tasks.rados.rados.0.smithi071.stdout:19: oid 19 version 6 is already newer than 5 2023-12-15T22:56:41.910 INFO:tasks.rados.rados.0.smithi071.stdout:19: oid 19 version 6 is already newer than 4 2023-12-15T22:56:41.910 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 6 (ObjNum 18 snap 0 seq_num 18) dirty exists 2023-12-15T22:56:41.910 INFO:tasks.rados.rados.0.smithi071.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2023-12-15T22:56:41.910 INFO:tasks.rados.rados.0.smithi071.stdout:13: finishing write tid 3 to smithi07183189-13 2023-12-15T22:56:41.910 INFO:tasks.rados.rados.0.smithi071.stdout:19: done (15 left) 2023-12-15T22:56:41.910 INFO:tasks.rados.rados.0.smithi071.stdout:30: write initial oid 30 2023-12-15T22:56:41.910 INFO:tasks.rados.rados.0.smithi071.stdout:30: seq_num 29 ranges {487403=531280,1514213=478671} 2023-12-15T22:56:41.916 INFO:tasks.rados.rados.0.smithi071.stdout:30: writing smithi07183189-30 from 487403 to 1018683 tid 1 2023-12-15T22:56:41.924 INFO:tasks.rados.rados.0.smithi071.stdout:30: writing smithi07183189-30 from 1514213 to 1992884 tid 2 2023-12-15T22:56:41.924 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2023-12-15T22:56:41.924 INFO:tasks.rados.rados.0.smithi071.stdout:20: finishing write tid 3 to smithi07183189-20 2023-12-15T22:56:41.924 INFO:tasks.rados.rados.0.smithi071.stdout:20: finishing write tid 4 to smithi07183189-20 2023-12-15T22:56:41.924 INFO:tasks.rados.rados.0.smithi071.stdout:20: oid 20 updating version 0 to 4 2023-12-15T22:56:41.924 INFO:tasks.rados.rados.0.smithi071.stdout:20: oid 20 updating version 4 to 6 2023-12-15T22:56:41.924 INFO:tasks.rados.rados.0.smithi071.stdout:20: oid 20 version 6 is already newer than 5 2023-12-15T22:56:41.924 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 20 v 6 (ObjNum 19 snap 0 seq_num 19) dirty exists 2023-12-15T22:56:41.924 INFO:tasks.rados.rados.0.smithi071.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2023-12-15T22:56:41.925 INFO:tasks.rados.rados.0.smithi071.stdout:13: finishing write tid 4 to smithi07183189-13 2023-12-15T22:56:41.925 INFO:tasks.rados.rados.0.smithi071.stdout:13: oid 13 updating version 0 to 6 2023-12-15T22:56:41.925 INFO:tasks.rados.rados.0.smithi071.stdout:13: oid 13 version 6 is already newer than 4 2023-12-15T22:56:41.925 INFO:tasks.rados.rados.0.smithi071.stdout:13: oid 13 version 6 is already newer than 6 2023-12-15T22:56:41.925 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 6 (ObjNum 12 snap 0 seq_num 12) dirty exists 2023-12-15T22:56:41.925 INFO:tasks.rados.rados.0.smithi071.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2023-12-15T22:56:41.925 INFO:tasks.rados.rados.0.smithi071.stdout:11: finishing write tid 2 to smithi07183189-11 2023-12-15T22:56:41.925 INFO:tasks.rados.rados.0.smithi071.stdout:11: finishing write tid 3 to smithi07183189-11 2023-12-15T22:56:41.925 INFO:tasks.rados.rados.0.smithi071.stdout:13: done (15 left) 2023-12-15T22:56:41.925 INFO:tasks.rados.rados.0.smithi071.stdout:20: done (14 left) 2023-12-15T22:56:41.925 INFO:tasks.rados.rados.0.smithi071.stdout:31: write initial oid 31 2023-12-15T22:56:41.925 INFO:tasks.rados.rados.0.smithi071.stdout:31: seq_num 30 ranges {504210=606529,1572695=622329} 2023-12-15T22:56:41.932 INFO:tasks.rados.rados.0.smithi071.stdout:31: writing smithi07183189-31 from 504210 to 1110739 tid 1 2023-12-15T22:56:41.940 INFO:tasks.rados.rados.0.smithi071.stdout:31: writing smithi07183189-31 from 1572695 to 2195024 tid 2 2023-12-15T22:56:41.940 INFO:tasks.rados.rados.0.smithi071.stdout:32: write initial oid 32 2023-12-15T22:56:41.940 INFO:tasks.rados.rados.0.smithi071.stdout:32: seq_num 31 ranges {521017=798131,1747530=482340} 2023-12-15T22:56:41.949 INFO:tasks.rados.rados.0.smithi071.stdout:32: writing smithi07183189-32 from 521017 to 1319148 tid 1 2023-12-15T22:56:41.956 INFO:tasks.rados.rados.0.smithi071.stdout:32: writing smithi07183189-32 from 1747530 to 2229870 tid 2 2023-12-15T22:56:41.957 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2023-12-15T22:56:41.957 INFO:tasks.rados.rados.0.smithi071.stdout:27: finishing write tid 1 to smithi07183189-27 2023-12-15T22:56:41.957 INFO:tasks.rados.rados.0.smithi071.stdout:11: finishing write tid 4 to smithi07183189-11 2023-12-15T22:56:41.957 INFO:tasks.rados.rados.0.smithi071.stdout:11: oid 11 updating version 0 to 4 2023-12-15T22:56:41.957 INFO:tasks.rados.rados.0.smithi071.stdout:11: oid 11 updating version 4 to 6 2023-12-15T22:56:41.957 INFO:tasks.rados.rados.0.smithi071.stdout:11: oid 11 version 6 is already newer than 6 2023-12-15T22:56:41.957 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 11 v 6 (ObjNum 10 snap 0 seq_num 10) dirty exists 2023-12-15T22:56:41.957 INFO:tasks.rados.rados.0.smithi071.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2023-12-15T22:56:41.957 INFO:tasks.rados.rados.0.smithi071.stdout:24: finishing write tid 3 to smithi07183189-24 2023-12-15T22:56:41.957 INFO:tasks.rados.rados.0.smithi071.stdout:11: done (15 left) 2023-12-15T22:56:41.957 INFO:tasks.rados.rados.0.smithi071.stdout:33: write initial oid 33 2023-12-15T22:56:41.957 INFO:tasks.rados.rados.0.smithi071.stdout:33: seq_num 32 ranges {537824=473380,1806012=625998} 2023-12-15T22:56:41.964 INFO:tasks.rados.rados.0.smithi071.stdout:33: writing smithi07183189-33 from 537824 to 1011204 tid 1 2023-12-15T22:56:41.975 INFO:tasks.rados.rados.0.smithi071.stdout:33: writing smithi07183189-33 from 1806012 to 2432010 tid 2 2023-12-15T22:56:41.975 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2023-12-15T22:56:41.975 INFO:tasks.rados.rados.0.smithi071.stdout:23: finishing write tid 3 to smithi07183189-23 2023-12-15T22:56:41.975 INFO:tasks.rados.rados.0.smithi071.stdout:23: finishing write tid 4 to smithi07183189-23 2023-12-15T22:56:41.975 INFO:tasks.rados.rados.0.smithi071.stdout:23: oid 23 updating version 0 to 1 2023-12-15T22:56:41.975 INFO:tasks.rados.rados.0.smithi071.stdout:23: oid 23 updating version 1 to 2 2023-12-15T22:56:41.975 INFO:tasks.rados.rados.0.smithi071.stdout:23: oid 23 updating version 2 to 3 2023-12-15T22:56:41.975 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 23 v 3 (ObjNum 22 snap 0 seq_num 22) dirty exists 2023-12-15T22:56:41.976 INFO:tasks.rados.rados.0.smithi071.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2023-12-15T22:56:41.976 INFO:tasks.rados.rados.0.smithi071.stdout:24: finishing write tid 4 to smithi07183189-24 2023-12-15T22:56:41.976 INFO:tasks.rados.rados.0.smithi071.stdout:24: oid 24 updating version 0 to 2 2023-12-15T22:56:41.976 INFO:tasks.rados.rados.0.smithi071.stdout:24: oid 24 updating version 2 to 3 2023-12-15T22:56:41.976 INFO:tasks.rados.rados.0.smithi071.stdout:24: oid 24 version 3 is already newer than 1 2023-12-15T22:56:41.976 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 3 (ObjNum 23 snap 0 seq_num 23) dirty exists 2023-12-15T22:56:41.976 INFO:tasks.rados.rados.0.smithi071.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2023-12-15T22:56:41.976 INFO:tasks.rados.rados.0.smithi071.stdout:23: done (15 left) 2023-12-15T22:56:41.976 INFO:tasks.rados.rados.0.smithi071.stdout:24: done (14 left) 2023-12-15T22:56:41.976 INFO:tasks.rados.rados.0.smithi071.stdout:34: write initial oid 34 2023-12-15T22:56:41.976 INFO:tasks.rados.rados.0.smithi071.stdout:34: seq_num 33 ranges {554631=548629,1748141=486009} 2023-12-15T22:56:41.982 INFO:tasks.rados.rados.0.smithi071.stdout:34: writing smithi07183189-34 from 554631 to 1103260 tid 1 2023-12-15T22:56:41.990 INFO:tasks.rados.rados.0.smithi071.stdout:34: writing smithi07183189-34 from 1748141 to 2234150 tid 2 2023-12-15T22:56:41.990 INFO:tasks.rados.rados.0.smithi071.stdout:35: write initial oid 35 2023-12-15T22:56:41.991 INFO:tasks.rados.rados.0.smithi071.stdout:35: seq_num 34 ranges {571438=623878,1806623=629667} 2023-12-15T22:56:42.000 INFO:tasks.rados.rados.0.smithi071.stdout:35: writing smithi07183189-35 from 571438 to 1195316 tid 1 2023-12-15T22:56:42.009 INFO:tasks.rados.rados.0.smithi071.stdout:35: writing smithi07183189-35 from 1806623 to 2436290 tid 2 2023-12-15T22:56:42.009 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2023-12-15T22:56:42.009 INFO:tasks.rados.rados.0.smithi071.stdout:27: finishing write tid 2 to smithi07183189-27 2023-12-15T22:56:42.009 INFO:tasks.rados.rados.0.smithi071.stdout:14: finishing write tid 1 to smithi07183189-14 2023-12-15T22:56:42.009 INFO:tasks.rados.rados.0.smithi071.stdout:16: finishing write tid 1 to smithi07183189-16 2023-12-15T22:56:42.009 INFO:tasks.rados.rados.0.smithi071.stdout:29: finishing write tid 1 to smithi07183189-29 2023-12-15T22:56:42.009 INFO:tasks.rados.rados.0.smithi071.stdout:15: finishing write tid 1 to smithi07183189-15 2023-12-15T22:56:42.009 INFO:tasks.rados.rados.0.smithi071.stdout:28: finishing write tid 1 to smithi07183189-28 2023-12-15T22:56:42.009 INFO:tasks.rados.rados.0.smithi071.stdout:30: finishing write tid 1 to smithi07183189-30 2023-12-15T22:56:42.026 INFO:tasks.rados.rados.0.smithi071.stdout:34: finishing write tid 1 to smithi07183189-34 2023-12-15T22:56:42.037 INFO:tasks.rados.rados.0.smithi071.stdout:27: finishing write tid 3 to smithi07183189-27 2023-12-15T22:56:42.040 INFO:tasks.rados.rados.0.smithi071.stdout:27: finishing write tid 4 to smithi07183189-27 2023-12-15T22:56:42.040 INFO:tasks.rados.rados.0.smithi071.stdout:27: oid 27 updating version 0 to 4 2023-12-15T22:56:42.040 INFO:tasks.rados.rados.0.smithi071.stdout:27: oid 27 updating version 4 to 6 2023-12-15T22:56:42.040 INFO:tasks.rados.rados.0.smithi071.stdout:27: oid 27 version 6 is already newer than 5 2023-12-15T22:56:42.040 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 6 (ObjNum 26 snap 0 seq_num 26) dirty exists 2023-12-15T22:56:42.041 INFO:tasks.rados.rados.0.smithi071.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2023-12-15T22:56:42.041 INFO:tasks.rados.rados.0.smithi071.stdout:27: done (15 left) 2023-12-15T22:56:42.041 INFO:tasks.rados.rados.0.smithi071.stdout:36: write initial oid 36 2023-12-15T22:56:42.041 INFO:tasks.rados.rados.0.smithi071.stdout:36: seq_num 35 ranges {588245=699127,1865105=489678} 2023-12-15T22:56:42.049 INFO:tasks.rados.rados.0.smithi071.stdout:36: writing smithi07183189-36 from 588245 to 1287372 tid 1 2023-12-15T22:56:42.058 INFO:tasks.rados.rados.0.smithi071.stdout:36: writing smithi07183189-36 from 1865105 to 2354783 tid 2 2023-12-15T22:56:42.058 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2023-12-15T22:56:42.058 INFO:tasks.rados.rados.0.smithi071.stdout:26: finishing write tid 1 to smithi07183189-26 2023-12-15T22:56:42.058 INFO:tasks.rados.rados.0.smithi071.stdout:14: finishing write tid 2 to smithi07183189-14 2023-12-15T22:56:42.058 INFO:tasks.rados.rados.0.smithi071.stdout:30: finishing write tid 2 to smithi07183189-30 2023-12-15T22:56:42.063 INFO:tasks.rados.rados.0.smithi071.stdout:14: finishing write tid 3 to smithi07183189-14 2023-12-15T22:56:42.066 INFO:tasks.rados.rados.0.smithi071.stdout:14: finishing write tid 4 to smithi07183189-14 2023-12-15T22:56:42.067 INFO:tasks.rados.rados.0.smithi071.stdout:14: oid 14 updating version 0 to 9 2023-12-15T22:56:42.067 INFO:tasks.rados.rados.0.smithi071.stdout:14: oid 14 version 9 is already newer than 9 2023-12-15T22:56:42.067 INFO:tasks.rados.rados.0.smithi071.stdout:14: oid 14 version 9 is already newer than 7 2023-12-15T22:56:42.067 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 9 (ObjNum 13 snap 0 seq_num 13) dirty exists 2023-12-15T22:56:42.067 INFO:tasks.rados.rados.0.smithi071.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2023-12-15T22:56:42.067 INFO:tasks.rados.rados.0.smithi071.stdout:14: done (15 left) 2023-12-15T22:56:42.067 INFO:tasks.rados.rados.0.smithi071.stdout:37: write initial oid 37 2023-12-15T22:56:42.067 INFO:tasks.rados.rados.0.smithi071.stdout:37: seq_num 36 ranges {605052=774376,1923587=633336} 2023-12-15T22:56:42.069 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:56:42.077 INFO:tasks.rados.rados.0.smithi071.stdout:37: writing smithi07183189-37 from 605052 to 1379428 tid 1 2023-12-15T22:56:42.086 INFO:tasks.rados.rados.0.smithi071.stdout:37: writing smithi07183189-37 from 1923587 to 2556923 tid 2 2023-12-15T22:56:42.086 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2023-12-15T22:56:42.086 INFO:tasks.rados.rados.0.smithi071.stdout:34: finishing write tid 2 to smithi07183189-34 2023-12-15T22:56:42.086 INFO:tasks.rados.rados.0.smithi071.stdout:16: finishing write tid 2 to smithi07183189-16 2023-12-15T22:56:42.086 INFO:tasks.rados.rados.0.smithi071.stdout:28: finishing write tid 2 to smithi07183189-28 2023-12-15T22:56:42.086 INFO:tasks.rados.rados.0.smithi071.stdout:29: finishing write tid 2 to smithi07183189-29 2023-12-15T22:56:42.089 INFO:tasks.rados.rados.0.smithi071.stdout:34: finishing write tid 3 to smithi07183189-34 2023-12-15T22:56:42.089 INFO:tasks.rados.rados.0.smithi071.stdout:15: finishing write tid 2 to smithi07183189-15 2023-12-15T22:56:42.090 INFO:tasks.rados.rados.0.smithi071.stdout:34: finishing write tid 4 to smithi07183189-34 2023-12-15T22:56:42.090 INFO:tasks.rados.rados.0.smithi071.stdout:34: oid 34 updating version 0 to 1 2023-12-15T22:56:42.090 INFO:tasks.rados.rados.0.smithi071.stdout:34: oid 34 updating version 1 to 3 2023-12-15T22:56:42.090 INFO:tasks.rados.rados.0.smithi071.stdout:34: oid 34 version 3 is already newer than 3 2023-12-15T22:56:42.090 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 3 (ObjNum 33 snap 0 seq_num 33) dirty exists 2023-12-15T22:56:42.090 INFO:tasks.rados.rados.0.smithi071.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2023-12-15T22:56:42.091 INFO:tasks.rados.rados.0.smithi071.stdout:34: done (15 left) 2023-12-15T22:56:42.091 INFO:tasks.rados.rados.0.smithi071.stdout:38: write initial oid 38 2023-12-15T22:56:42.091 INFO:tasks.rados.rados.0.smithi071.stdout:38: seq_num 37 ranges {621859=449625,1865716=756143} 2023-12-15T22:56:42.097 INFO:tasks.rados.rados.0.smithi071.stdout:38: writing smithi07183189-38 from 621859 to 1071484 tid 1 2023-12-15T22:56:42.109 INFO:tasks.rados.rados.0.smithi071.stdout:38: writing smithi07183189-38 from 1865716 to 2621859 tid 2 2023-12-15T22:56:42.109 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2023-12-15T22:56:42.109 INFO:tasks.rados.rados.0.smithi071.stdout:29: finishing write tid 3 to smithi07183189-29 2023-12-15T22:56:42.109 INFO:tasks.rados.rados.0.smithi071.stdout:29: finishing write tid 4 to smithi07183189-29 2023-12-15T22:56:42.109 INFO:tasks.rados.rados.0.smithi071.stdout:29: oid 29 updating version 0 to 7 2023-12-15T22:56:42.109 INFO:tasks.rados.rados.0.smithi071.stdout:29: oid 29 updating version 7 to 9 2023-12-15T22:56:42.109 INFO:tasks.rados.rados.0.smithi071.stdout:29: oid 29 version 9 is already newer than 9 2023-12-15T22:56:42.109 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 9 (ObjNum 28 snap 0 seq_num 28) dirty exists 2023-12-15T22:56:42.110 INFO:tasks.rados.rados.0.smithi071.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2023-12-15T22:56:42.110 INFO:tasks.rados.rados.0.smithi071.stdout:29: done (15 left) 2023-12-15T22:56:42.110 INFO:tasks.rados.rados.0.smithi071.stdout:39: write initial oid 39 2023-12-15T22:56:42.110 INFO:tasks.rados.rados.0.smithi071.stdout:39: seq_num 38 ranges {638666=524874,1924198=637005} 2023-12-15T22:56:42.117 INFO:tasks.rados.rados.0.smithi071.stdout:39: writing smithi07183189-39 from 638666 to 1163540 tid 1 2023-12-15T22:56:42.126 INFO:tasks.rados.rados.0.smithi071.stdout:39: writing smithi07183189-39 from 1924198 to 2561203 tid 2 2023-12-15T22:56:42.126 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2023-12-15T22:56:42.126 INFO:tasks.rados.rados.0.smithi071.stdout:15: finishing write tid 3 to smithi07183189-15 2023-12-15T22:56:42.126 INFO:tasks.rados.rados.0.smithi071.stdout:15: finishing write tid 4 to smithi07183189-15 2023-12-15T22:56:42.126 INFO:tasks.rados.rados.0.smithi071.stdout:15: oid 15 updating version 0 to 9 2023-12-15T22:56:42.126 INFO:tasks.rados.rados.0.smithi071.stdout:15: oid 15 version 9 is already newer than 7 2023-12-15T22:56:42.126 INFO:tasks.rados.rados.0.smithi071.stdout:15: oid 15 version 9 is already newer than 9 2023-12-15T22:56:42.126 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 9 (ObjNum 14 snap 0 seq_num 14) dirty exists 2023-12-15T22:56:42.126 INFO:tasks.rados.rados.0.smithi071.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2023-12-15T22:56:42.126 INFO:tasks.rados.rados.0.smithi071.stdout:32: finishing write tid 1 to smithi07183189-32 2023-12-15T22:56:42.126 INFO:tasks.rados.rados.0.smithi071.stdout:15: done (15 left) 2023-12-15T22:56:42.126 INFO:tasks.rados.rados.0.smithi071.stdout:40: write initial oid 40 2023-12-15T22:56:42.127 INFO:tasks.rados.rados.0.smithi071.stdout:40: seq_num 39 ranges {655473=716476,2099033=556440} 2023-12-15T22:56:42.133 INFO:tasks.rados.rados.0.smithi071.stdout:40: writing smithi07183189-40 from 655473 to 1371949 tid 1 2023-12-15T22:56:42.141 INFO:tasks.rados.rados.0.smithi071.stdout:40: writing smithi07183189-40 from 2099033 to 2655473 tid 2 2023-12-15T22:56:42.141 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2023-12-15T22:56:42.141 INFO:tasks.rados.rados.0.smithi071.stdout:26: finishing write tid 2 to smithi07183189-26 2023-12-15T22:56:42.156 INFO:tasks.rados.rados.0.smithi071.stdout:30: finishing write tid 3 to smithi07183189-30 2023-12-15T22:56:42.157 INFO:tasks.rados.rados.0.smithi071.stdout:30: finishing write tid 4 to smithi07183189-30 2023-12-15T22:56:42.157 INFO:tasks.rados.rados.0.smithi071.stdout:30: oid 30 updating version 0 to 3 2023-12-15T22:56:42.158 INFO:tasks.rados.rados.0.smithi071.stdout:30: oid 30 version 3 is already newer than 2 2023-12-15T22:56:42.158 INFO:tasks.rados.rados.0.smithi071.stdout:30: oid 30 version 3 is already newer than 1 2023-12-15T22:56:42.158 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 3 (ObjNum 29 snap 0 seq_num 29) dirty exists 2023-12-15T22:56:42.158 INFO:tasks.rados.rados.0.smithi071.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2023-12-15T22:56:42.158 INFO:tasks.rados.rados.0.smithi071.stdout:30: done (15 left) 2023-12-15T22:56:42.158 INFO:tasks.rados.rados.0.smithi071.stdout:41: write initial oid 41 2023-12-15T22:56:42.158 INFO:tasks.rados.rados.0.smithi071.stdout:41: seq_num 40 ranges {672280=791725,2157515=514765} 2023-12-15T22:56:42.166 INFO:tasks.rados.rados.0.smithi071.stdout:41: writing smithi07183189-41 from 672280 to 1464005 tid 1 2023-12-15T22:56:42.174 INFO:tasks.rados.rados.0.smithi071.stdout:41: writing smithi07183189-41 from 2157515 to 2672280 tid 2 2023-12-15T22:56:42.174 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2023-12-15T22:56:42.174 INFO:tasks.rados.rados.0.smithi071.stdout:28: finishing write tid 3 to smithi07183189-28 2023-12-15T22:56:42.174 INFO:tasks.rados.rados.0.smithi071.stdout:28: finishing write tid 4 to smithi07183189-28 2023-12-15T22:56:42.174 INFO:tasks.rados.rados.0.smithi071.stdout:28: oid 28 updating version 0 to 4 2023-12-15T22:56:42.174 INFO:tasks.rados.rados.0.smithi071.stdout:28: oid 28 updating version 4 to 6 2023-12-15T22:56:42.174 INFO:tasks.rados.rados.0.smithi071.stdout:28: oid 28 version 6 is already newer than 5 2023-12-15T22:56:42.175 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 6 (ObjNum 27 snap 0 seq_num 27) dirty exists 2023-12-15T22:56:42.175 INFO:tasks.rados.rados.0.smithi071.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2023-12-15T22:56:42.175 INFO:tasks.rados.rados.0.smithi071.stdout:28: done (15 left) 2023-12-15T22:56:42.175 INFO:tasks.rados.rados.0.smithi071.stdout:42: write initial oid 42 2023-12-15T22:56:42.175 INFO:tasks.rados.rados.0.smithi071.stdout:42: seq_num 41 ranges {689087=466974,1699644=784332} 2023-12-15T22:56:42.180 INFO:tasks.rados.rados.0.smithi071.stdout:42: writing smithi07183189-42 from 689087 to 1156061 tid 1 2023-12-15T22:56:42.185 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:42.183+0000 7ffa12d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:42.188 INFO:tasks.rados.rados.0.smithi071.stdout:42: writing smithi07183189-42 from 1699644 to 2483976 tid 2 2023-12-15T22:56:42.188 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2023-12-15T22:56:42.188 INFO:tasks.rados.rados.0.smithi071.stdout:21: finishing write tid 1 to smithi07183189-21 2023-12-15T22:56:42.188 INFO:tasks.rados.rados.0.smithi071.stdout:33: finishing write tid 1 to smithi07183189-33 2023-12-15T22:56:42.188 INFO:tasks.rados.rados.0.smithi071.stdout:39: finishing write tid 1 to smithi07183189-39 2023-12-15T22:56:42.189 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:42.189+0000 7ffa12d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:42.189 INFO:tasks.rados.rados.0.smithi071.stdout:35: finishing write tid 1 to smithi07183189-35 2023-12-15T22:56:42.191 INFO:tasks.rados.rados.0.smithi071.stdout:16: finishing write tid 3 to smithi07183189-16 2023-12-15T22:56:42.196 INFO:tasks.rados.rados.0.smithi071.stdout:16: finishing write tid 4 to smithi07183189-16 2023-12-15T22:56:42.196 INFO:tasks.rados.rados.0.smithi071.stdout:16: oid 16 updating version 0 to 8 2023-12-15T22:56:42.196 INFO:tasks.rados.rados.0.smithi071.stdout:16: oid 16 updating version 8 to 9 2023-12-15T22:56:42.196 INFO:tasks.rados.rados.0.smithi071.stdout:16: oid 16 version 9 is already newer than 7 2023-12-15T22:56:42.197 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 16 v 9 (ObjNum 15 snap 0 seq_num 15) dirty exists 2023-12-15T22:56:42.197 INFO:tasks.rados.rados.0.smithi071.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2023-12-15T22:56:42.197 INFO:tasks.rados.rados.0.smithi071.stdout:16: done (15 left) 2023-12-15T22:56:42.197 INFO:tasks.rados.rados.0.smithi071.stdout:43: write initial oid 43 2023-12-15T22:56:42.197 INFO:tasks.rados.rados.0.smithi071.stdout:43: seq_num 42 ranges {705894=542223,1758126=644343} 2023-12-15T22:56:42.204 INFO:tasks.rados.rados.0.smithi071.stdout:43: writing smithi07183189-43 from 705894 to 1248117 tid 1 2023-12-15T22:56:42.211 INFO:tasks.rados.rados.0.smithi071.stdout:43: writing smithi07183189-43 from 1758126 to 2402469 tid 2 2023-12-15T22:56:42.211 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2023-12-15T22:56:42.211 INFO:tasks.rados.rados.0.smithi071.stdout:38: finishing write tid 1 to smithi07183189-38 2023-12-15T22:56:42.221 INFO:tasks.rados.rados.0.smithi071.stdout:39: finishing write tid 2 to smithi07183189-39 2023-12-15T22:56:42.226 INFO:tasks.rados.rados.0.smithi071.stdout:26: finishing write tid 3 to smithi07183189-26 2023-12-15T22:56:42.226 INFO:tasks.rados.rados.0.smithi071.stdout:26: finishing write tid 4 to smithi07183189-26 2023-12-15T22:56:42.227 INFO:tasks.rados.rados.0.smithi071.stdout:26: oid 26 updating version 0 to 4 2023-12-15T22:56:42.227 INFO:tasks.rados.rados.0.smithi071.stdout:26: oid 26 updating version 4 to 6 2023-12-15T22:56:42.227 INFO:tasks.rados.rados.0.smithi071.stdout:26: oid 26 version 6 is already newer than 5 2023-12-15T22:56:42.227 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 6 (ObjNum 25 snap 0 seq_num 25) dirty exists 2023-12-15T22:56:42.227 INFO:tasks.rados.rados.0.smithi071.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2023-12-15T22:56:42.227 INFO:tasks.rados.rados.0.smithi071.stdout:26: done (15 left) 2023-12-15T22:56:42.227 INFO:tasks.rados.rados.0.smithi071.stdout:44: write initial oid 44 2023-12-15T22:56:42.227 INFO:tasks.rados.rados.0.smithi071.stdout:44: seq_num 43 ranges {722701=617472,1816608=788001} 2023-12-15T22:56:42.234 INFO:tasks.rados.rados.0.smithi071.stdout:44: writing smithi07183189-44 from 722701 to 1340173 tid 1 2023-12-15T22:56:42.242 INFO:tasks.rados.rados.0.smithi071.stdout:44: writing smithi07183189-44 from 1816608 to 2604609 tid 2 2023-12-15T22:56:42.242 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2023-12-15T22:56:42.242 INFO:tasks.rados.rados.0.smithi071.stdout:32: finishing write tid 2 to smithi07183189-32 2023-12-15T22:56:42.243 INFO:tasks.rados.rados.0.smithi071.stdout:39: finishing write tid 3 to smithi07183189-39 2023-12-15T22:56:42.243 INFO:tasks.rados.rados.0.smithi071.stdout:39: finishing write tid 4 to smithi07183189-39 2023-12-15T22:56:42.243 INFO:tasks.rados.rados.0.smithi071.stdout:39: oid 39 updating version 0 to 7 2023-12-15T22:56:42.243 INFO:tasks.rados.rados.0.smithi071.stdout:39: oid 39 updating version 7 to 9 2023-12-15T22:56:42.243 INFO:tasks.rados.rados.0.smithi071.stdout:39: oid 39 version 9 is already newer than 9 2023-12-15T22:56:42.243 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 39 v 9 (ObjNum 38 snap 0 seq_num 38) dirty exists 2023-12-15T22:56:42.243 INFO:tasks.rados.rados.0.smithi071.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2023-12-15T22:56:42.243 INFO:tasks.rados.rados.0.smithi071.stdout:39: done (15 left) 2023-12-15T22:56:42.243 INFO:tasks.rados.rados.0.smithi071.stdout:45: write initial oid 45 2023-12-15T22:56:42.243 INFO:tasks.rados.rados.0.smithi071.stdout:45: seq_num 44 ranges {739508=692721,1875090=648012} 2023-12-15T22:56:42.251 INFO:tasks.rados.rados.0.smithi071.stdout:45: writing smithi07183189-45 from 739508 to 1432229 tid 1 2023-12-15T22:56:42.259 INFO:tasks.rados.rados.0.smithi071.stdout:45: writing smithi07183189-45 from 1875090 to 2523102 tid 2 2023-12-15T22:56:42.259 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2023-12-15T22:56:42.263 INFO:tasks.rados.rados.0.smithi071.stdout:17: finishing write tid 1 to smithi07183189-17 2023-12-15T22:56:42.275 INFO:tasks.rados.rados.0.smithi071.stdout:33: finishing write tid 2 to smithi07183189-33 2023-12-15T22:56:42.276 INFO:tasks.rados.rados.0.smithi071.stdout:21: finishing write tid 2 to smithi07183189-21 2023-12-15T22:56:42.277 INFO:tasks.rados.rados.0.smithi071.stdout:21: finishing write tid 3 to smithi07183189-21 2023-12-15T22:56:42.279 INFO:tasks.rados.rados.0.smithi071.stdout:21: finishing write tid 4 to smithi07183189-21 2023-12-15T22:56:42.279 INFO:tasks.rados.rados.0.smithi071.stdout:21: oid 21 updating version 0 to 12 2023-12-15T22:56:42.279 INFO:tasks.rados.rados.0.smithi071.stdout:21: oid 21 version 12 is already newer than 10 2023-12-15T22:56:42.279 INFO:tasks.rados.rados.0.smithi071.stdout:21: oid 21 version 12 is already newer than 12 2023-12-15T22:56:42.279 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 12 (ObjNum 20 snap 0 seq_num 20) dirty exists 2023-12-15T22:56:42.279 INFO:tasks.rados.rados.0.smithi071.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2023-12-15T22:56:42.279 INFO:tasks.rados.rados.0.smithi071.stdout:21: done (15 left) 2023-12-15T22:56:42.279 INFO:tasks.rados.rados.0.smithi071.stdout:46: write initial oid 46 2023-12-15T22:56:42.280 INFO:tasks.rados.rados.0.smithi071.stdout:46: seq_num 45 ranges {756315=767970,1933572=791670} 2023-12-15T22:56:42.288 INFO:tasks.rados.rados.0.smithi071.stdout:46: writing smithi07183189-46 from 756315 to 1524285 tid 1 2023-12-15T22:56:42.295 INFO:tasks.rados.rados.0.smithi071.stdout:46: writing smithi07183189-46 from 1933572 to 2725242 tid 2 2023-12-15T22:56:42.295 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2023-12-15T22:56:42.295 INFO:tasks.rados.rados.0.smithi071.stdout:38: finishing write tid 2 to smithi07183189-38 2023-12-15T22:56:42.295 INFO:tasks.rados.rados.0.smithi071.stdout:31: finishing write tid 1 to smithi07183189-31 2023-12-15T22:56:42.295 INFO:tasks.rados.rados.0.smithi071.stdout:35: finishing write tid 2 to smithi07183189-35 2023-12-15T22:56:42.295 INFO:tasks.rados.rados.0.smithi071.stdout:42: finishing write tid 1 to smithi07183189-42 2023-12-15T22:56:42.299 INFO:tasks.rados.rados.0.smithi071.stdout:43: finishing write tid 1 to smithi07183189-43 2023-12-15T22:56:42.302 INFO:tasks.rados.rados.0.smithi071.stdout:32: finishing write tid 3 to smithi07183189-32 2023-12-15T22:56:42.303 INFO:tasks.rados.rados.0.smithi071.stdout:44: finishing write tid 1 to smithi07183189-44 2023-12-15T22:56:42.314 INFO:tasks.rados.rados.0.smithi071.stdout:32: finishing write tid 4 to smithi07183189-32 2023-12-15T22:56:42.314 INFO:tasks.rados.rados.0.smithi071.stdout:32: oid 32 updating version 0 to 5 2023-12-15T22:56:42.314 INFO:tasks.rados.rados.0.smithi071.stdout:32: oid 32 updating version 5 to 6 2023-12-15T22:56:42.314 INFO:tasks.rados.rados.0.smithi071.stdout:32: oid 32 version 6 is already newer than 4 2023-12-15T22:56:42.315 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 6 (ObjNum 31 snap 0 seq_num 31) dirty exists 2023-12-15T22:56:42.315 INFO:tasks.rados.rados.0.smithi071.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2023-12-15T22:56:42.315 INFO:tasks.rados.rados.0.smithi071.stdout:32: done (15 left) 2023-12-15T22:56:42.315 INFO:tasks.rados.rados.0.smithi071.stdout:47: write initial oid 47 2023-12-15T22:56:42.315 INFO:tasks.rados.rados.0.smithi071.stdout:47: seq_num 46 ranges {773122=559572,1992054=651681} 2023-12-15T22:56:42.322 INFO:tasks.rados.rados.0.smithi071.stdout:47: writing smithi07183189-47 from 773122 to 1332694 tid 1 2023-12-15T22:56:42.329 INFO:tasks.rados.rados.0.smithi071.stdout:47: writing smithi07183189-47 from 1992054 to 2643735 tid 2 2023-12-15T22:56:42.330 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2023-12-15T22:56:42.335 INFO:tasks.rados.rados.0.smithi071.stdout:38: finishing write tid 3 to smithi07183189-38 2023-12-15T22:56:42.335 INFO:tasks.rados.rados.0.smithi071.stdout:38: finishing write tid 4 to smithi07183189-38 2023-12-15T22:56:42.335 INFO:tasks.rados.rados.0.smithi071.stdout:38: oid 38 updating version 0 to 8 2023-12-15T22:56:42.335 INFO:tasks.rados.rados.0.smithi071.stdout:38: oid 38 updating version 8 to 9 2023-12-15T22:56:42.335 INFO:tasks.rados.rados.0.smithi071.stdout:38: oid 38 version 9 is already newer than 7 2023-12-15T22:56:42.335 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 38 v 9 (ObjNum 37 snap 0 seq_num 37) dirty exists 2023-12-15T22:56:42.335 INFO:tasks.rados.rados.0.smithi071.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2023-12-15T22:56:42.335 INFO:tasks.rados.rados.0.smithi071.stdout:38: done (15 left) 2023-12-15T22:56:42.335 INFO:tasks.rados.rados.0.smithi071.stdout:48: write initial oid 48 2023-12-15T22:56:42.335 INFO:tasks.rados.rados.0.smithi071.stdout:48: seq_num 47 ranges {789929=634821,2050536=739393} 2023-12-15T22:56:42.342 INFO:tasks.rados.rados.0.smithi071.stdout:48: writing smithi07183189-48 from 789929 to 1424750 tid 1 2023-12-15T22:56:42.350 INFO:tasks.rados.rados.0.smithi071.stdout:48: writing smithi07183189-48 from 2050536 to 2789929 tid 2 2023-12-15T22:56:42.350 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2023-12-15T22:56:42.350 INFO:tasks.rados.rados.0.smithi071.stdout:17: finishing write tid 2 to smithi07183189-17 2023-12-15T22:56:42.365 INFO:tasks.rados.rados.0.smithi071.stdout:17: finishing write tid 3 to smithi07183189-17 2023-12-15T22:56:42.367 INFO:tasks.rados.rados.0.smithi071.stdout:41: finishing write tid 1 to smithi07183189-41 2023-12-15T22:56:42.369 INFO:tasks.rados.rados.0.smithi071.stdout:17: finishing write tid 4 to smithi07183189-17 2023-12-15T22:56:42.369 INFO:tasks.rados.rados.0.smithi071.stdout:17: oid 17 updating version 0 to 10 2023-12-15T22:56:42.369 INFO:tasks.rados.rados.0.smithi071.stdout:17: oid 17 updating version 10 to 12 2023-12-15T22:56:42.369 INFO:tasks.rados.rados.0.smithi071.stdout:17: oid 17 version 12 is already newer than 12 2023-12-15T22:56:42.369 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 17 v 12 (ObjNum 16 snap 0 seq_num 16) dirty exists 2023-12-15T22:56:42.369 INFO:tasks.rados.rados.0.smithi071.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2023-12-15T22:56:42.369 INFO:tasks.rados.rados.0.smithi071.stdout:17: done (15 left) 2023-12-15T22:56:42.369 INFO:tasks.rados.rados.0.smithi071.stdout:49: write initial oid 49 2023-12-15T22:56:42.369 INFO:tasks.rados.rados.0.smithi071.stdout:49: seq_num 48 ranges {406736=710070,1709018=655350} 2023-12-15T22:56:42.375 INFO:tasks.rados.rados.0.smithi071.stdout:49: writing smithi07183189-49 from 406736 to 1116806 tid 1 2023-12-15T22:56:42.384 INFO:tasks.rados.rados.0.smithi071.stdout:49: writing smithi07183189-49 from 1709018 to 2364368 tid 2 2023-12-15T22:56:42.384 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2023-12-15T22:56:42.384 INFO:tasks.rados.rados.0.smithi071.stdout:43: finishing write tid 2 to smithi07183189-43 2023-12-15T22:56:42.384 INFO:tasks.rados.rados.0.smithi071.stdout:42: finishing write tid 2 to smithi07183189-42 2023-12-15T22:56:42.389 INFO:tasks.rados.rados.0.smithi071.stdout:44: finishing write tid 2 to smithi07183189-44 2023-12-15T22:56:42.390 INFO:tasks.rados.rados.0.smithi071.stdout:35: finishing write tid 3 to smithi07183189-35 2023-12-15T22:56:42.393 INFO:tasks.rados.rados.0.smithi071.stdout:33: finishing write tid 3 to smithi07183189-33 2023-12-15T22:56:42.394 INFO:tasks.rados.rados.0.smithi071.stdout:33: finishing write tid 4 to smithi07183189-33 2023-12-15T22:56:42.394 INFO:tasks.rados.rados.0.smithi071.stdout:33: oid 33 updating version 0 to 12 2023-12-15T22:56:42.394 INFO:tasks.rados.rados.0.smithi071.stdout:33: oid 33 version 12 is already newer than 10 2023-12-15T22:56:42.394 INFO:tasks.rados.rados.0.smithi071.stdout:33: oid 33 version 12 is already newer than 11 2023-12-15T22:56:42.394 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 33 v 12 (ObjNum 32 snap 0 seq_num 32) dirty exists 2023-12-15T22:56:42.394 INFO:tasks.rados.rados.0.smithi071.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2023-12-15T22:56:42.394 INFO:tasks.rados.rados.0.smithi071.stdout:33: done (15 left) 2023-12-15T22:56:42.394 INFO:tasks.rados.rados.0.smithi071.stdout:50: write initial oid 50 2023-12-15T22:56:42.394 INFO:tasks.rados.rados.0.smithi071.stdout:50: seq_num 49 ranges {423543=785319,1767500=799008} 2023-12-15T22:56:42.400 INFO:tasks.rados.rados.0.smithi071.stdout:50: writing smithi07183189-50 from 423543 to 1208862 tid 1 2023-12-15T22:56:42.409 INFO:tasks.rados.rados.0.smithi071.stdout:50: writing smithi07183189-50 from 1767500 to 2566508 tid 2 2023-12-15T22:56:42.409 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2023-12-15T22:56:42.409 INFO:tasks.rados.rados.0.smithi071.stdout:44: finishing write tid 3 to smithi07183189-44 2023-12-15T22:56:42.409 INFO:tasks.rados.rados.0.smithi071.stdout:44: finishing write tid 4 to smithi07183189-44 2023-12-15T22:56:42.409 INFO:tasks.rados.rados.0.smithi071.stdout:44: oid 44 updating version 0 to 3 2023-12-15T22:56:42.410 INFO:tasks.rados.rados.0.smithi071.stdout:44: oid 44 version 3 is already newer than 3 2023-12-15T22:56:42.410 INFO:tasks.rados.rados.0.smithi071.stdout:44: oid 44 version 3 is already newer than 1 2023-12-15T22:56:42.410 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 3 (ObjNum 43 snap 0 seq_num 43) dirty exists 2023-12-15T22:56:42.410 INFO:tasks.rados.rados.0.smithi071.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2023-12-15T22:56:42.410 INFO:tasks.rados.rados.0.smithi071.stdout:31: finishing write tid 2 to smithi07183189-31 2023-12-15T22:56:42.410 INFO:tasks.rados.rados.0.smithi071.stdout:42: finishing write tid 3 to smithi07183189-42 2023-12-15T22:56:42.410 INFO:tasks.rados.rados.0.smithi071.stdout:44: done (15 left) 2023-12-15T22:56:42.410 INFO:tasks.rados.rados.0.smithi071.stdout:51: write (excl) oid 2 current snap is 0 2023-12-15T22:56:42.410 INFO:tasks.rados.rados.0.smithi071.stdout:42: finishing write tid 4 to smithi07183189-42 2023-12-15T22:56:42.410 INFO:tasks.rados.rados.0.smithi071.stdout:42: oid 42 updating version 0 to 10 2023-12-15T22:56:42.410 INFO:tasks.rados.rados.0.smithi071.stdout:42: oid 42 updating version 10 to 12 2023-12-15T22:56:42.410 INFO:tasks.rados.rados.0.smithi071.stdout:42: oid 42 version 12 is already newer than 12 2023-12-15T22:56:42.410 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 12 (ObjNum 41 snap 0 seq_num 41) dirty exists 2023-12-15T22:56:42.411 INFO:tasks.rados.rados.0.smithi071.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2023-12-15T22:56:42.411 INFO:tasks.rados.rados.0.smithi071.stdout:35: finishing write tid 4 to smithi07183189-35 2023-12-15T22:56:42.411 INFO:tasks.rados.rados.0.smithi071.stdout:35: oid 35 updating version 0 to 7 2023-12-15T22:56:42.411 INFO:tasks.rados.rados.0.smithi071.stdout:35: oid 35 updating version 7 to 9 2023-12-15T22:56:42.411 INFO:tasks.rados.rados.0.smithi071.stdout:35: oid 35 version 9 is already newer than 8 2023-12-15T22:56:42.411 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 9 (ObjNum 34 snap 0 seq_num 34) dirty exists 2023-12-15T22:56:42.411 INFO:tasks.rados.rados.0.smithi071.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2023-12-15T22:56:42.411 INFO:tasks.rados.rados.0.smithi071.stdout:51: seq_num 50 ranges {440350=460568,1309629=542666,2323973=516377} 2023-12-15T22:56:42.413 INFO:tasks.rados.rados.0.smithi071.stdout:51: writing smithi07183189-2 from 440350 to 900918 tid 1 2023-12-15T22:56:42.419 INFO:tasks.rados.rados.0.smithi071.stdout:51: writing smithi07183189-2 from 1309629 to 1852295 tid 2 2023-12-15T22:56:42.425 INFO:tasks.rados.rados.0.smithi071.stdout:51: writing smithi07183189-2 from 2323973 to 2840350 tid 3 2023-12-15T22:56:42.425 INFO:tasks.rados.rados.0.smithi071.stdout:35: done (15 left) 2023-12-15T22:56:42.425 INFO:tasks.rados.rados.0.smithi071.stdout:42: done (14 left) 2023-12-15T22:56:42.425 INFO:tasks.rados.rados.0.smithi071.stdout:52: write (excl) oid 16 current snap is 0 2023-12-15T22:56:42.425 INFO:tasks.rados.rados.0.smithi071.stdout:47: finishing write tid 1 to smithi07183189-47 2023-12-15T22:56:42.426 INFO:tasks.rados.rados.0.smithi071.stdout:52: seq_num 51 ranges {457157=535817,1768111=402677} 2023-12-15T22:56:42.430 INFO:tasks.rados.rados.0.smithi071.stdout:52: writing smithi07183189-16 from 457157 to 992974 tid 1 2023-12-15T22:56:42.438 INFO:tasks.rados.rados.0.smithi071.stdout:52: writing smithi07183189-16 from 1768111 to 2170788 tid 2 2023-12-15T22:56:42.438 INFO:tasks.rados.rados.0.smithi071.stdout:53: read oid 20 snap -1 2023-12-15T22:56:42.438 INFO:tasks.rados.rados.0.smithi071.stdout:53: expect (ObjNum 19 snap 0 seq_num 19) 2023-12-15T22:56:42.438 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2023-12-15T22:56:42.438 INFO:tasks.rados.rados.0.smithi071.stdout:49: finishing write tid 1 to smithi07183189-49 2023-12-15T22:56:42.438 INFO:tasks.rados.rados.0.smithi071.stdout:43: finishing write tid 3 to smithi07183189-43 2023-12-15T22:56:42.438 INFO:tasks.rados.rados.0.smithi071.stdout:43: finishing write tid 4 to smithi07183189-43 2023-12-15T22:56:42.438 INFO:tasks.rados.rados.0.smithi071.stdout:43: oid 43 updating version 0 to 6 2023-12-15T22:56:42.438 INFO:tasks.rados.rados.0.smithi071.stdout:43: oid 43 version 6 is already newer than 5 2023-12-15T22:56:42.438 INFO:tasks.rados.rados.0.smithi071.stdout:43: oid 43 version 6 is already newer than 4 2023-12-15T22:56:42.439 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 6 (ObjNum 42 snap 0 seq_num 42) dirty exists 2023-12-15T22:56:42.439 INFO:tasks.rados.rados.0.smithi071.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2023-12-15T22:56:42.439 INFO:tasks.rados.rados.0.smithi071.stdout:43: done (15 left) 2023-12-15T22:56:42.439 INFO:tasks.rados.rados.0.smithi071.stdout:54: delete oid 23 current snap is 0 2023-12-15T22:56:42.439 INFO:tasks.rados.rados.0.smithi071.stdout:25: finishing write tid 1 to smithi07183189-25 2023-12-15T22:56:42.456 INFO:tasks.rados.rados.0.smithi071.stdout:41: finishing write tid 2 to smithi07183189-41 2023-12-15T22:56:42.460 INFO:tasks.rados.rados.0.smithi071.stdout:46: finishing write tid 1 to smithi07183189-46 2023-12-15T22:56:42.467 INFO:tasks.rados.rados.0.smithi071.stdout:37: finishing write tid 1 to smithi07183189-37 2023-12-15T22:56:42.478 INFO:tasks.rados.rados.0.smithi071.stdout:41: finishing write tid 3 to smithi07183189-41 2023-12-15T22:56:42.481 INFO:tasks.rados.rados.0.smithi071.stdout:41: finishing write tid 4 to smithi07183189-41 2023-12-15T22:56:42.481 INFO:tasks.rados.rados.0.smithi071.stdout:41: oid 41 updating version 0 to 15 2023-12-15T22:56:42.481 INFO:tasks.rados.rados.0.smithi071.stdout:41: oid 41 version 15 is already newer than 13 2023-12-15T22:56:42.481 INFO:tasks.rados.rados.0.smithi071.stdout:41: oid 41 version 15 is already newer than 15 2023-12-15T22:56:42.481 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 41 v 15 (ObjNum 40 snap 0 seq_num 40) dirty exists 2023-12-15T22:56:42.481 INFO:tasks.rados.rados.0.smithi071.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2023-12-15T22:56:42.482 INFO:tasks.rados.rados.0.smithi071.stdout:31: finishing write tid 3 to smithi07183189-31 2023-12-15T22:56:42.487 INFO:tasks.rados.rados.0.smithi071.stdout:48: finishing write tid 1 to smithi07183189-48 2023-12-15T22:56:42.500 INFO:tasks.rados.rados.0.smithi071.stdout:31: finishing write tid 4 to smithi07183189-31 2023-12-15T22:56:42.500 INFO:tasks.rados.rados.0.smithi071.stdout:31: oid 31 updating version 0 to 9 2023-12-15T22:56:42.500 INFO:tasks.rados.rados.0.smithi071.stdout:31: oid 31 version 9 is already newer than 8 2023-12-15T22:56:42.501 INFO:tasks.rados.rados.0.smithi071.stdout:31: oid 31 version 9 is already newer than 7 2023-12-15T22:56:42.501 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 9 (ObjNum 30 snap 0 seq_num 30) dirty exists 2023-12-15T22:56:42.501 INFO:tasks.rados.rados.0.smithi071.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2023-12-15T22:56:42.512 INFO:tasks.rados.rados.0.smithi071.stdout:49: finishing write tid 2 to smithi07183189-49 2023-12-15T22:56:42.517 INFO:tasks.rados.rados.0.smithi071.stdout:47: finishing write tid 2 to smithi07183189-47 2023-12-15T22:56:42.519 INFO:tasks.rados.rados.0.smithi071.stdout:49: finishing write tid 3 to smithi07183189-49 2023-12-15T22:56:42.520 INFO:tasks.rados.rados.0.smithi071.stdout:25: finishing write tid 2 to smithi07183189-25 2023-12-15T22:56:42.520 INFO:tasks.rados.rados.0.smithi071.stdout:49: finishing write tid 4 to smithi07183189-49 2023-12-15T22:56:42.521 INFO:tasks.rados.rados.0.smithi071.stdout:49: oid 49 updating version 0 to 12 2023-12-15T22:56:42.521 INFO:tasks.rados.rados.0.smithi071.stdout:49: oid 49 version 12 is already newer than 12 2023-12-15T22:56:42.521 INFO:tasks.rados.rados.0.smithi071.stdout:49: oid 49 version 12 is already newer than 10 2023-12-15T22:56:42.521 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 49 v 12 (ObjNum 48 snap 0 seq_num 48) dirty exists 2023-12-15T22:56:42.521 INFO:tasks.rados.rados.0.smithi071.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2023-12-15T22:56:42.526 INFO:tasks.rados.rados.0.smithi071.stdout:25: finishing write tid 3 to smithi07183189-25 2023-12-15T22:56:42.529 INFO:tasks.rados.rados.0.smithi071.stdout:25: finishing write tid 4 to smithi07183189-25 2023-12-15T22:56:42.529 INFO:tasks.rados.rados.0.smithi071.stdout:25: oid 25 updating version 0 to 13 2023-12-15T22:56:42.529 INFO:tasks.rados.rados.0.smithi071.stdout:25: oid 25 updating version 13 to 15 2023-12-15T22:56:42.529 INFO:tasks.rados.rados.0.smithi071.stdout:25: oid 25 version 15 is already newer than 15 2023-12-15T22:56:42.529 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 25 v 15 (ObjNum 24 snap 0 seq_num 24) dirty exists 2023-12-15T22:56:42.530 INFO:tasks.rados.rados.0.smithi071.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2023-12-15T22:56:42.537 INFO:tasks.rados.rados.0.smithi071.stdout:25: done (15 left) 2023-12-15T22:56:42.537 INFO:tasks.rados.rados.0.smithi071.stdout:31: done (14 left) 2023-12-15T22:56:42.537 INFO:tasks.rados.rados.0.smithi071.stdout:41: done (13 left) 2023-12-15T22:56:42.537 INFO:tasks.rados.rados.0.smithi071.stdout:49: done (12 left) 2023-12-15T22:56:42.537 INFO:tasks.rados.rados.0.smithi071.stdout:54: done (11 left) 2023-12-15T22:56:42.537 INFO:tasks.rados.rados.0.smithi071.stdout:55: write oid 9 current snap is 0 2023-12-15T22:56:42.537 INFO:tasks.rados.rados.0.smithi071.stdout:55: seq_num 53 ranges {490771=686315,1885075=406346} 2023-12-15T22:56:42.543 INFO:tasks.rados.rados.0.smithi071.stdout:55: writing smithi07183189-9 from 490771 to 1177086 tid 1 2023-12-15T22:56:42.550 INFO:tasks.rados.rados.0.smithi071.stdout:55: writing smithi07183189-9 from 1885075 to 2291421 tid 2 2023-12-15T22:56:42.550 INFO:tasks.rados.rados.0.smithi071.stdout:56: write oid 38 current snap is 0 2023-12-15T22:56:42.550 INFO:tasks.rados.rados.0.smithi071.stdout:56: seq_num 54 ranges {507578=477917,1543557=550004,2833665=73913} 2023-12-15T22:56:42.555 INFO:tasks.rados.rados.0.smithi071.stdout:56: writing smithi07183189-38 from 507578 to 985495 tid 1 2023-12-15T22:56:42.561 INFO:tasks.rados.rados.0.smithi071.stdout:56: writing smithi07183189-38 from 1543557 to 2093561 tid 2 2023-12-15T22:56:42.566 INFO:tasks.rados.rados.0.smithi071.stdout:56: writing smithi07183189-38 from 2833665 to 2907578 tid 3 2023-12-15T22:56:42.566 INFO:tasks.rados.rados.0.smithi071.stdout:57: write (excl) oid 21 current snap is 0 2023-12-15T22:56:42.566 INFO:tasks.rados.rados.0.smithi071.stdout:57: seq_num 55 ranges {524385=553166,1602039=410015,2577441=346944} 2023-12-15T22:56:42.572 INFO:tasks.rados.rados.0.smithi071.stdout:57: writing smithi07183189-21 from 524385 to 1077551 tid 1 2023-12-15T22:56:42.577 INFO:tasks.rados.rados.0.smithi071.stdout:57: writing smithi07183189-21 from 1602039 to 2012054 tid 2 2023-12-15T22:56:42.584 INFO:tasks.rados.rados.0.smithi071.stdout:57: writing smithi07183189-21 from 2577441 to 2924385 tid 3 2023-12-15T22:56:42.584 INFO:tasks.rados.rados.0.smithi071.stdout:46: finishing write tid 2 to smithi07183189-46 2023-12-15T22:56:42.584 INFO:tasks.rados.rados.0.smithi071.stdout:58: read oid 8 snap -1 2023-12-15T22:56:42.584 INFO:tasks.rados.rados.0.smithi071.stdout:58: expect (ObjNum 7 snap 0 seq_num 7) 2023-12-15T22:56:42.584 INFO:tasks.rados.rados.0.smithi071.stdout:59: write (excl) oid 24 current snap is 0 2023-12-15T22:56:42.585 INFO:tasks.rados.rados.0.smithi071.stdout:37: finishing write tid 2 to smithi07183189-37 2023-12-15T22:56:42.585 INFO:tasks.rados.rados.0.smithi071.stdout:47: finishing write tid 3 to smithi07183189-47 2023-12-15T22:56:42.585 INFO:tasks.rados.rados.0.smithi071.stdout:47: finishing write tid 4 to smithi07183189-47 2023-12-15T22:56:42.585 INFO:tasks.rados.rados.0.smithi071.stdout:47: oid 47 updating version 0 to 7 2023-12-15T22:56:42.585 INFO:tasks.rados.rados.0.smithi071.stdout:47: oid 47 updating version 7 to 8 2023-12-15T22:56:42.585 INFO:tasks.rados.rados.0.smithi071.stdout:47: oid 47 updating version 8 to 9 2023-12-15T22:56:42.585 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 9 (ObjNum 46 snap 0 seq_num 46) dirty exists 2023-12-15T22:56:42.585 INFO:tasks.rados.rados.0.smithi071.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2023-12-15T22:56:42.585 INFO:tasks.rados.rados.0.smithi071.stdout:37: finishing write tid 3 to smithi07183189-37 2023-12-15T22:56:42.585 INFO:tasks.rados.rados.0.smithi071.stdout:48: finishing write tid 2 to smithi07183189-48 2023-12-15T22:56:42.585 INFO:tasks.rados.rados.0.smithi071.stdout:56: finishing write tid 1 to smithi07183189-38 2023-12-15T22:56:42.585 INFO:tasks.rados.rados.0.smithi071.stdout:37: finishing write tid 4 to smithi07183189-37 2023-12-15T22:56:42.585 INFO:tasks.rados.rados.0.smithi071.stdout:37: oid 37 updating version 0 to 13 2023-12-15T22:56:42.585 INFO:tasks.rados.rados.0.smithi071.stdout:37: oid 37 updating version 13 to 15 2023-12-15T22:56:42.585 INFO:tasks.rados.rados.0.smithi071.stdout:37: oid 37 version 15 is already newer than 15 2023-12-15T22:56:42.586 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 15 (ObjNum 36 snap 0 seq_num 36) dirty exists 2023-12-15T22:56:42.586 INFO:tasks.rados.rados.0.smithi071.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2023-12-15T22:56:42.586 INFO:tasks.rados.rados.0.smithi071.stdout:51: finishing write tid 1 to smithi07183189-2 2023-12-15T22:56:42.586 INFO:tasks.rados.rados.0.smithi071.stdout:59: seq_num 56 ranges {541192=628415,1660521=553673,2888511=52681} 2023-12-15T22:56:42.590 INFO:tasks.rados.rados.0.smithi071.stdout:59: writing smithi07183189-24 from 541192 to 1169607 tid 1 2023-12-15T22:56:42.596 INFO:tasks.rados.rados.0.smithi071.stdout:59: writing smithi07183189-24 from 1660521 to 2214194 tid 2 2023-12-15T22:56:42.600 INFO:tasks.rados.rados.0.smithi071.stdout:59: writing smithi07183189-24 from 2888511 to 2941192 tid 3 2023-12-15T22:56:42.600 INFO:tasks.rados.rados.0.smithi071.stdout:37: done (15 left) 2023-12-15T22:56:42.601 INFO:tasks.rados.rados.0.smithi071.stdout:47: done (14 left) 2023-12-15T22:56:42.601 INFO:tasks.rados.rados.0.smithi071.stdout:60: delete oid 31 current snap is 0 2023-12-15T22:56:42.602 INFO:tasks.rados.rados.0.smithi071.stdout:46: finishing write tid 3 to smithi07183189-46 2023-12-15T22:56:42.606 INFO:tasks.rados.rados.0.smithi071.stdout:46: finishing write tid 4 to smithi07183189-46 2023-12-15T22:56:42.606 INFO:tasks.rados.rados.0.smithi071.stdout:46: oid 46 updating version 0 to 5 2023-12-15T22:56:42.606 INFO:tasks.rados.rados.0.smithi071.stdout:46: oid 46 updating version 5 to 6 2023-12-15T22:56:42.606 INFO:tasks.rados.rados.0.smithi071.stdout:46: oid 46 version 6 is already newer than 4 2023-12-15T22:56:42.606 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 6 (ObjNum 45 snap 0 seq_num 45) dirty exists 2023-12-15T22:56:42.607 INFO:tasks.rados.rados.0.smithi071.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2023-12-15T22:56:42.629 INFO:tasks.rados.rados.0.smithi071.stdout:56: finishing write tid 2 to smithi07183189-38 2023-12-15T22:56:42.629 INFO:tasks.rados.rados.0.smithi071.stdout:36: finishing write tid 1 to smithi07183189-36 2023-12-15T22:56:42.637 INFO:tasks.rados.rados.0.smithi071.stdout:40: finishing write tid 1 to smithi07183189-40 2023-12-15T22:56:42.639 INFO:tasks.rados.rados.0.smithi071.stdout:57: finishing write tid 1 to smithi07183189-21 2023-12-15T22:56:42.651 INFO:tasks.rados.rados.0.smithi071.stdout:51: finishing write tid 2 to smithi07183189-2 2023-12-15T22:56:42.654 INFO:tasks.rados.rados.0.smithi071.stdout:51: finishing write tid 3 to smithi07183189-2 2023-12-15T22:56:42.659 INFO:tasks.rados.rados.0.smithi071.stdout:48: finishing write tid 3 to smithi07183189-48 2023-12-15T22:56:42.659 INFO:tasks.rados.rados.0.smithi071.stdout:48: finishing write tid 4 to smithi07183189-48 2023-12-15T22:56:42.659 INFO:tasks.rados.rados.0.smithi071.stdout:48: oid 48 updating version 0 to 14 2023-12-15T22:56:42.659 INFO:tasks.rados.rados.0.smithi071.stdout:48: oid 48 version 14 is already newer than 13 2023-12-15T22:56:42.659 INFO:tasks.rados.rados.0.smithi071.stdout:48: oid 48 updating version 14 to 15 2023-12-15T22:56:42.659 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 15 (ObjNum 47 snap 0 seq_num 47) dirty exists 2023-12-15T22:56:42.659 INFO:tasks.rados.rados.0.smithi071.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2023-12-15T22:56:42.660 INFO:tasks.rados.rados.0.smithi071.stdout:59: finishing write tid 1 to smithi07183189-24 2023-12-15T22:56:42.662 INFO:tasks.rados.rados.0.smithi071.stdout:46: done (14 left) 2023-12-15T22:56:42.662 INFO:tasks.rados.rados.0.smithi071.stdout:48: done (13 left) 2023-12-15T22:56:42.662 INFO:tasks.rados.rados.0.smithi071.stdout:60: done (12 left) 2023-12-15T22:56:42.662 INFO:tasks.rados.rados.0.smithi071.stdout:61: delete oid 13 current snap is 0 2023-12-15T22:56:42.665 INFO:tasks.rados.rados.0.smithi071.stdout:55: finishing write tid 1 to smithi07183189-9 2023-12-15T22:56:42.691 INFO:tasks.rados.rados.0.smithi071.stdout:56: finishing write tid 3 to smithi07183189-38 2023-12-15T22:56:42.693 INFO:tasks.rados.rados.0.smithi071.stdout:57: finishing write tid 2 to smithi07183189-21 2023-12-15T22:56:42.696 INFO:tasks.rados.rados.0.smithi071.stdout:59: finishing write tid 2 to smithi07183189-24 2023-12-15T22:56:42.704 INFO:tasks.rados.rados.0.smithi071.stdout:40: finishing write tid 2 to smithi07183189-40 2023-12-15T22:56:42.704 INFO:tasks.rados.rados.0.smithi071.stdout:55: finishing write tid 2 to smithi07183189-9 2023-12-15T22:56:42.706 INFO:tasks.rados.rados.0.smithi071.stdout:40: finishing write tid 3 to smithi07183189-40 2023-12-15T22:56:42.708 INFO:tasks.rados.rados.0.smithi071.stdout:51: finishing write tid 4 to smithi07183189-2 2023-12-15T22:56:42.711 INFO:tasks.rados.rados.0.smithi071.stdout:36: finishing write tid 2 to smithi07183189-36 2023-12-15T22:56:42.713 INFO:tasks.rados.rados.0.smithi071.stdout:36: finishing write tid 3 to smithi07183189-36 2023-12-15T22:56:42.721 INFO:tasks.rados.rados.0.smithi071.stdout:40: finishing write tid 4 to smithi07183189-40 2023-12-15T22:56:42.722 INFO:tasks.rados.rados.0.smithi071.stdout:40: oid 40 updating version 0 to 16 2023-12-15T22:56:42.722 INFO:tasks.rados.rados.0.smithi071.stdout:40: oid 40 updating version 16 to 18 2023-12-15T22:56:42.722 INFO:tasks.rados.rados.0.smithi071.stdout:40: oid 40 version 18 is already newer than 18 2023-12-15T22:56:42.722 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 40 v 18 (ObjNum 39 snap 0 seq_num 39) dirty exists 2023-12-15T22:56:42.722 INFO:tasks.rados.rados.0.smithi071.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2023-12-15T22:56:42.722 INFO:tasks.rados.rados.0.smithi071.stdout:51: finishing write tid 5 to smithi07183189-2 2023-12-15T22:56:42.722 INFO:tasks.rados.rados.0.smithi071.stdout:51: oid 2 updating version 0 to 10 2023-12-15T22:56:42.722 INFO:tasks.rados.rados.0.smithi071.stdout:51: oid 2 version 10 is already newer than 9 2023-12-15T22:56:42.722 INFO:tasks.rados.rados.0.smithi071.stdout:51: oid 2 version 10 is already newer than 9 2023-12-15T22:56:42.722 INFO:tasks.rados.rados.0.smithi071.stdout:51: oid 2 version 10 is already newer than 7 2023-12-15T22:56:42.722 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 10 (ObjNum 50 snap 0 seq_num 50) dirty exists 2023-12-15T22:56:42.722 INFO:tasks.rados.rados.0.smithi071.stdout:51: left oid 2 (ObjNum 50 snap 0 seq_num 50) 2023-12-15T22:56:42.723 INFO:tasks.rados.rados.0.smithi071.stdout:55: finishing write tid 3 to smithi07183189-9 2023-12-15T22:56:42.723 INFO:tasks.rados.rados.0.smithi071.stdout:55: finishing write tid 4 to smithi07183189-9 2023-12-15T22:56:42.723 INFO:tasks.rados.rados.0.smithi071.stdout:55: oid 9 updating version 0 to 12 2023-12-15T22:56:42.723 INFO:tasks.rados.rados.0.smithi071.stdout:55: oid 9 version 12 is already newer than 12 2023-12-15T22:56:42.723 INFO:tasks.rados.rados.0.smithi071.stdout:55: oid 9 version 12 is already newer than 10 2023-12-15T22:56:42.723 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 12 (ObjNum 53 snap 0 seq_num 53) dirty exists 2023-12-15T22:56:42.724 INFO:tasks.rados.rados.0.smithi071.stdout:55: left oid 9 (ObjNum 53 snap 0 seq_num 53) 2023-12-15T22:56:42.725 INFO:tasks.rados.rados.0.smithi071.stdout:36: finishing write tid 4 to smithi07183189-36 2023-12-15T22:56:42.725 INFO:tasks.rados.rados.0.smithi071.stdout:36: oid 36 updating version 0 to 18 2023-12-15T22:56:42.725 INFO:tasks.rados.rados.0.smithi071.stdout:36: oid 36 version 18 is already newer than 18 2023-12-15T22:56:42.725 INFO:tasks.rados.rados.0.smithi071.stdout:36: oid 36 version 18 is already newer than 16 2023-12-15T22:56:42.725 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 18 (ObjNum 35 snap 0 seq_num 35) dirty exists 2023-12-15T22:56:42.725 INFO:tasks.rados.rados.0.smithi071.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2023-12-15T22:56:42.728 INFO:tasks.rados.rados.0.smithi071.stdout:56: finishing write tid 4 to smithi07183189-38 2023-12-15T22:56:42.732 INFO:tasks.rados.rados.0.smithi071.stdout:57: finishing write tid 3 to smithi07183189-21 2023-12-15T22:56:42.735 INFO:tasks.rados.rados.0.smithi071.stdout:56: finishing write tid 5 to smithi07183189-38 2023-12-15T22:56:42.735 INFO:tasks.rados.rados.0.smithi071.stdout:56: oid 38 updating version 0 to 13 2023-12-15T22:56:42.735 INFO:tasks.rados.rados.0.smithi071.stdout:56: oid 38 version 13 is already newer than 11 2023-12-15T22:56:42.735 INFO:tasks.rados.rados.0.smithi071.stdout:56: oid 38 version 13 is already newer than 10 2023-12-15T22:56:42.735 INFO:tasks.rados.rados.0.smithi071.stdout:56: oid 38 version 13 is already newer than 12 2023-12-15T22:56:42.735 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 38 v 13 (ObjNum 54 snap 0 seq_num 54) dirty exists 2023-12-15T22:56:42.735 INFO:tasks.rados.rados.0.smithi071.stdout:56: left oid 38 (ObjNum 54 snap 0 seq_num 54) 2023-12-15T22:56:42.740 INFO:tasks.rados.rados.0.smithi071.stdout:59: finishing write tid 3 to smithi07183189-24 2023-12-15T22:56:42.748 INFO:tasks.rados.rados.0.smithi071.stdout:52: finishing write tid 1 to smithi07183189-16 2023-12-15T22:56:42.749 INFO:tasks.rados.rados.0.smithi071.stdout:45: finishing write tid 1 to smithi07183189-45 2023-12-15T22:56:42.770 INFO:tasks.rados.rados.0.smithi071.stdout:57: finishing write tid 4 to smithi07183189-21 2023-12-15T22:56:42.772 INFO:tasks.rados.rados.0.smithi071.stdout:59: finishing write tid 4 to smithi07183189-24 2023-12-15T22:56:42.773 INFO:tasks.rados.rados.0.smithi071.stdout:57: finishing write tid 5 to smithi07183189-21 2023-12-15T22:56:42.774 INFO:tasks.rados.rados.0.smithi071.stdout:57: oid 21 updating version 0 to 16 2023-12-15T22:56:42.774 INFO:tasks.rados.rados.0.smithi071.stdout:57: oid 21 updating version 16 to 17 2023-12-15T22:56:42.774 INFO:tasks.rados.rados.0.smithi071.stdout:57: oid 21 updating version 17 to 18 2023-12-15T22:56:42.774 INFO:tasks.rados.rados.0.smithi071.stdout:57: oid 21 updating version 18 to 19 2023-12-15T22:56:42.774 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 19 (ObjNum 55 snap 0 seq_num 55) dirty exists 2023-12-15T22:56:42.774 INFO:tasks.rados.rados.0.smithi071.stdout:57: left oid 21 (ObjNum 55 snap 0 seq_num 55) 2023-12-15T22:56:42.775 INFO:tasks.rados.rados.0.smithi071.stdout:59: finishing write tid 5 to smithi07183189-24 2023-12-15T22:56:42.775 INFO:tasks.rados.rados.0.smithi071.stdout:59: oid 24 updating version 0 to 4 2023-12-15T22:56:42.775 INFO:tasks.rados.rados.0.smithi071.stdout:59: oid 24 updating version 4 to 5 2023-12-15T22:56:42.775 INFO:tasks.rados.rados.0.smithi071.stdout:59: oid 24 updating version 5 to 7 2023-12-15T22:56:42.775 INFO:tasks.rados.rados.0.smithi071.stdout:59: oid 24 version 7 is already newer than 6 2023-12-15T22:56:42.775 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 7 (ObjNum 56 snap 0 seq_num 56) dirty exists 2023-12-15T22:56:42.775 INFO:tasks.rados.rados.0.smithi071.stdout:59: left oid 24 (ObjNum 56 snap 0 seq_num 56) 2023-12-15T22:56:42.778 INFO:tasks.rados.rados.0.smithi071.stdout:45: finishing write tid 2 to smithi07183189-45 2023-12-15T22:56:42.779 INFO:tasks.rados.rados.0.smithi071.stdout:45: finishing write tid 3 to smithi07183189-45 2023-12-15T22:56:42.781 INFO:tasks.rados.rados.0.smithi071.stdout:52: finishing write tid 2 to smithi07183189-16 2023-12-15T22:56:42.782 INFO:tasks.rados.rados.0.smithi071.stdout:58: expect (ObjNum 7 snap 0 seq_num 7) 2023-12-15T22:56:42.880 INFO:tasks.rados.rados.0.smithi071.stdout:45: finishing write tid 4 to smithi07183189-45 2023-12-15T22:56:42.880 INFO:tasks.rados.rados.0.smithi071.stdout:45: oid 45 updating version 0 to 21 2023-12-15T22:56:42.880 INFO:tasks.rados.rados.0.smithi071.stdout:45: oid 45 version 21 is already newer than 19 2023-12-15T22:56:42.880 INFO:tasks.rados.rados.0.smithi071.stdout:45: oid 45 version 21 is already newer than 21 2023-12-15T22:56:42.880 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 21 (ObjNum 44 snap 0 seq_num 44) dirty exists 2023-12-15T22:56:42.880 INFO:tasks.rados.rados.0.smithi071.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2023-12-15T22:56:42.880 INFO:tasks.rados.rados.0.smithi071.stdout:52: finishing write tid 3 to smithi07183189-16 2023-12-15T22:56:42.881 INFO:tasks.rados.rados.0.smithi071.stdout:36: done (12 left) 2023-12-15T22:56:42.881 INFO:tasks.rados.rados.0.smithi071.stdout:40: done (11 left) 2023-12-15T22:56:42.881 INFO:tasks.rados.rados.0.smithi071.stdout:45: done (10 left) 2023-12-15T22:56:42.881 INFO:tasks.rados.rados.0.smithi071.stdout:51: done (9 left) 2023-12-15T22:56:42.881 INFO:tasks.rados.rados.0.smithi071.stdout:55: done (8 left) 2023-12-15T22:56:42.881 INFO:tasks.rados.rados.0.smithi071.stdout:56: done (7 left) 2023-12-15T22:56:42.881 INFO:tasks.rados.rados.0.smithi071.stdout:57: done (6 left) 2023-12-15T22:56:42.881 INFO:tasks.rados.rados.0.smithi071.stdout:58: done (5 left) 2023-12-15T22:56:42.881 INFO:tasks.rados.rados.0.smithi071.stdout:59: done (4 left) 2023-12-15T22:56:42.881 INFO:tasks.rados.rados.0.smithi071.stdout:61: done (3 left) 2023-12-15T22:56:42.881 INFO:tasks.rados.rados.0.smithi071.stdout:62: delete oid 47 current snap is 0 2023-12-15T22:56:42.881 INFO:tasks.rados.rados.0.smithi071.stdout:52: finishing write tid 4 to smithi07183189-16 2023-12-15T22:56:42.881 INFO:tasks.rados.rados.0.smithi071.stdout:52: oid 16 updating version 0 to 19 2023-12-15T22:56:42.881 INFO:tasks.rados.rados.0.smithi071.stdout:52: oid 16 updating version 19 to 21 2023-12-15T22:56:42.881 INFO:tasks.rados.rados.0.smithi071.stdout:52: oid 16 version 21 is already newer than 21 2023-12-15T22:56:42.882 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 16 v 21 (ObjNum 51 snap 0 seq_num 51) dirty exists 2023-12-15T22:56:42.882 INFO:tasks.rados.rados.0.smithi071.stdout:52: left oid 16 (ObjNum 51 snap 0 seq_num 51) 2023-12-15T22:56:42.882 INFO:tasks.rados.rados.0.smithi071.stdout:50: finishing write tid 1 to smithi07183189-50 2023-12-15T22:56:42.882 INFO:tasks.rados.rados.0.smithi071.stdout:50: finishing write tid 2 to smithi07183189-50 2023-12-15T22:56:42.882 INFO:tasks.rados.rados.0.smithi071.stdout:50: finishing write tid 3 to smithi07183189-50 2023-12-15T22:56:42.882 INFO:tasks.rados.rados.0.smithi071.stdout:50: finishing write tid 4 to smithi07183189-50 2023-12-15T22:56:42.882 INFO:tasks.rados.rados.0.smithi071.stdout:50: oid 50 updating version 0 to 24 2023-12-15T22:56:42.882 INFO:tasks.rados.rados.0.smithi071.stdout:50: oid 50 version 24 is already newer than 24 2023-12-15T22:56:42.882 INFO:tasks.rados.rados.0.smithi071.stdout:50: oid 50 version 24 is already newer than 22 2023-12-15T22:56:42.882 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 24 (ObjNum 49 snap 0 seq_num 49) dirty exists 2023-12-15T22:56:42.882 INFO:tasks.rados.rados.0.smithi071.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2023-12-15T22:56:42.882 INFO:tasks.rados.rados.0.smithi071.stdout:53: expect (ObjNum 19 snap 0 seq_num 19) 2023-12-15T22:56:42.973 INFO:tasks.rados.rados.0.smithi071.stdout:50: done (3 left) 2023-12-15T22:56:42.973 INFO:tasks.rados.rados.0.smithi071.stdout:52: done (2 left) 2023-12-15T22:56:42.973 INFO:tasks.rados.rados.0.smithi071.stdout:53: done (1 left) 2023-12-15T22:56:42.973 INFO:tasks.rados.rados.0.smithi071.stdout:62: done (0 left) 2023-12-15T22:56:42.973 INFO:tasks.rados.rados.0.smithi071.stdout:63: read oid 15 snap -1 2023-12-15T22:56:42.973 INFO:tasks.rados.rados.0.smithi071.stdout:63: expect (ObjNum 14 snap 0 seq_num 14) 2023-12-15T22:56:42.974 INFO:tasks.rados.rados.0.smithi071.stdout:64: snap_create 2023-12-15T22:56:42.987 INFO:tasks.rados.rados.0.smithi071.stdout:63: expect (ObjNum 14 snap 0 seq_num 14) 2023-12-15T22:56:43.222 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T22:56:43.222 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T22:56:43.222 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T22:56:43.327 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:43.323+0000 7fbc2699e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:43.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:43.330+0000 7fbc2699e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:43.378 INFO:tasks.rados.rados.0.smithi071.stdout:63: done (1 left) 2023-12-15T22:56:43.378 INFO:tasks.rados.rados.0.smithi071.stdout:64: done (0 left) 2023-12-15T22:56:43.378 INFO:tasks.rados.rados.0.smithi071.stdout:65: write oid 40 current snap is 1 2023-12-15T22:56:43.379 INFO:tasks.rados.rados.0.smithi071.stdout:65: seq_num 60 ranges {608420=529411,1778096=561011,2881850=126570} 2023-12-15T22:56:43.384 INFO:tasks.rados.rados.0.smithi071.stdout:65: writing smithi07183189-40 from 608420 to 1137831 tid 1 2023-12-15T22:56:43.391 INFO:tasks.rados.rados.0.smithi071.stdout:65: writing smithi07183189-40 from 1778096 to 2339107 tid 2 2023-12-15T22:56:43.396 INFO:tasks.rados.rados.0.smithi071.stdout:65: writing smithi07183189-40 from 2881850 to 3008420 tid 3 2023-12-15T22:56:43.396 INFO:tasks.rados.rados.0.smithi071.stdout:66: read oid 48 snap 0 2023-12-15T22:56:43.396 INFO:tasks.rados.rados.0.smithi071.stdout:66: expect (ObjNum 47 snap 0 seq_num 47) 2023-12-15T22:56:43.396 INFO:tasks.rados.rados.0.smithi071.stdout:67: read oid 28 snap 0 2023-12-15T22:56:43.396 INFO:tasks.rados.rados.0.smithi071.stdout:67: expect (ObjNum 27 snap 0 seq_num 27) 2023-12-15T22:56:43.396 INFO:tasks.rados.rados.0.smithi071.stdout:68: read oid 13 snap -1 2023-12-15T22:56:43.396 INFO:tasks.rados.rados.0.smithi071.stdout:68: expect deleted 2023-12-15T22:56:43.397 INFO:tasks.rados.rados.0.smithi071.stdout:69: read oid 43 snap -1 2023-12-15T22:56:43.397 INFO:tasks.rados.rados.0.smithi071.stdout:69: expect (ObjNum 42 snap 0 seq_num 42) 2023-12-15T22:56:43.397 INFO:tasks.rados.rados.0.smithi071.stdout:70: read oid 36 snap -1 2023-12-15T22:56:43.397 INFO:tasks.rados.rados.0.smithi071.stdout:70: expect (ObjNum 35 snap 0 seq_num 35) 2023-12-15T22:56:43.397 INFO:tasks.rados.rados.0.smithi071.stdout:71: read oid 50 snap -1 2023-12-15T22:56:43.397 INFO:tasks.rados.rados.0.smithi071.stdout:71: expect (ObjNum 49 snap 0 seq_num 49) 2023-12-15T22:56:43.397 INFO:tasks.rados.rados.0.smithi071.stdout:72: read oid 8 snap 0 2023-12-15T22:56:43.397 INFO:tasks.rados.rados.0.smithi071.stdout:72: expect (ObjNum 7 snap 0 seq_num 7) 2023-12-15T22:56:43.397 INFO:tasks.rados.rados.0.smithi071.stdout:73: read oid 27 snap 0 2023-12-15T22:56:43.397 INFO:tasks.rados.rados.0.smithi071.stdout:73: expect (ObjNum 26 snap 0 seq_num 26) 2023-12-15T22:56:43.397 INFO:tasks.rados.rados.0.smithi071.stdout:74: snap_create 2023-12-15T22:56:43.433 INFO:tasks.rados.rados.0.smithi071.stdout:70: expect (ObjNum 35 snap 0 seq_num 35) 2023-12-15T22:56:43.546 INFO:tasks.rados.rados.0.smithi071.stdout:66: expect (ObjNum 47 snap 0 seq_num 47) 2023-12-15T22:56:43.560 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:56:43.561 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-12-15T22:56:43.573 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2023-12-15T22:56:43.377350+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107732,"kb_used_data":436,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379628,"statfs":{"total":386547056640,"available":386436739072,"internally_reserved":0,"allocated":446464,"data_stored":1874680,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.864666"},"pg_stats":[{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-12-15T22:56:40.375024+0000","last_change":"2023-12-15T22:56:39.513156+0000","last_active":"2023-12-15T22:56:40.375024+0000","last_peered":"2023-12-15T22:56:40.375024+0000","last_clean":"2023-12-15T22:56:40.375024+0000","last_became_active":"2023-12-15T22:56:39.503797+0000","last_became_peered":"2023-12-15T22:56:39.503797+0000","last_unstale":"2023-12-15T22:56:40.375024+0000","last_undegraded":"2023-12-15T22:56:40.375024+0000","last_fullsized":"2023-12-15T22:56:40.375024+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":4,"reported_epoch":15,"state":"creating+activating","last_fresh":"2023-12-15T22:56:39.399490+0000","last_change":"2023-12-15T22:56:39.399490+0000","last_active":"2023-12-15T22:56:38.356579+0000","last_peered":"2023-12-15T22:56:38.356579+0000","last_clean":"2023-12-15T22:56:38.356579+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-15T22:56:39.399490+0000","last_undegraded":"2023-12-15T22:56:39.399490+0000","last_fullsized":"2023-12-15T22:56:39.399490+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":["2","1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-15T22:56:39.506904+0000","last_change":"2023-12-15T22:56:39.507322+0000","last_active":"2023-12-15T22:56:39.506904+0000","last_peered":"2023-12-15T22:56:39.506904+0000","last_clean":"2023-12-15T22:56:38.356579+0000","last_became_active":"2023-12-15T22:56:39.495490+0000","last_became_peered":"2023-12-15T22:56:39.495490+0000","last_unstale":"2023-12-15T22:56:39.506904+0000","last_undegraded":"2023-12-15T22:56:39.506904+0000","last_fullsized":"2023-12-15T22:56:39.506904+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-12-15T22:56:40.378971+0000","last_change":"2023-12-15T22:56:39.583080+0000","last_active":"2023-12-15T22:56:40.378971+0000","last_peered":"2023-12-15T22:56:40.378971+0000","last_clean":"2023-12-15T22:56:40.378971+0000","last_became_active":"2023-12-15T22:56:39.579234+0000","last_became_peered":"2023-12-15T22:56:39.579234+0000","last_unstale":"2023-12-15T22:56:40.378971+0000","last_undegraded":"2023-12-15T22:56:40.378971+0000","last_fullsized":"2023-12-15T22:56:40.378971+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-12-15T22:56:40.554653+0000","last_change":"2023-12-15T22:56:39.575062+0000","last_active":"2023-12-15T22:56:40.554653+0000","last_peered":"2023-12-15T22:56:40.554653+0000","last_clean":"2023-12-15T22:56:40.554653+0000","last_became_active":"2023-12-15T22:56:39.572176+0000","last_became_peered":"2023-12-15T22:56:39.572176+0000","last_unstale":"2023-12-15T22:56:40.554653+0000","last_undegraded":"2023-12-15T22:56:40.554653+0000","last_fullsized":"2023-12-15T22:56:40.554653+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-12-15T22:56:40.374041+0000","last_change":"2023-12-15T22:56:39.521391+0000","last_active":"2023-12-15T22:56:40.374041+0000","last_peered":"2023-12-15T22:56:40.374041+0000","last_clean":"2023-12-15T22:56:40.374041+0000","last_became_active":"2023-12-15T22:56:39.516361+0000","last_became_peered":"2023-12-15T22:56:39.516361+0000","last_unstale":"2023-12-15T22:56:40.374041+0000","last_undegraded":"2023-12-15T22:56:40.374041+0000","last_fullsized":"2023-12-15T22:56:40.374041+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-12-15T22:56:40.379067+0000","last_change":"2023-12-15T22:56:39.523120+0000","last_active":"2023-12-15T22:56:40.379067+0000","last_peered":"2023-12-15T22:56:40.379067+0000","last_clean":"2023-12-15T22:56:40.379067+0000","last_became_active":"2023-12-15T22:56:39.519748+0000","last_became_peered":"2023-12-15T22:56:39.519748+0000","last_unstale":"2023-12-15T22:56:40.379067+0000","last_undegraded":"2023-12-15T22:56:40.379067+0000","last_fullsized":"2023-12-15T22:56:40.379067+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-15T22:56:39.545699+0000","last_change":"2023-12-15T22:56:39.546051+0000","last_active":"2023-12-15T22:56:39.545699+0000","last_peered":"2023-12-15T22:56:39.545699+0000","last_clean":"2023-12-15T22:56:38.356579+0000","last_became_active":"2023-12-15T22:56:39.543179+0000","last_became_peered":"2023-12-15T22:56:39.543179+0000","last_unstale":"2023-12-15T22:56:39.545699+0000","last_undegraded":"2023-12-15T22:56:39.545699+0000","last_fullsized":"2023-12-15T22:56:39.545699+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":144,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-12-15T22:56:40.542052+0000","last_change":"2023-12-15T22:56:32.607017+0000","last_active":"2023-12-15T22:56:40.542052+0000","last_peered":"2023-12-15T22:56:40.542052+0000","last_clean":"2023-12-15T22:56:40.542052+0000","last_became_active":"2023-12-15T22:56:32.602384+0000","last_became_peered":"2023-12-15T22:56:32.602384+0000","last_unstale":"2023-12-15T22:56:40.542052+0000","last_undegraded":"2023-12-15T22:56:40.542052+0000","last_fullsized":"2023-12-15T22:56:40.542052+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_clean_scrub_stamp":"2023-12-15T22:56:31.525572+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-15T22:56:39.385630+0000","last_change":"2023-12-15T22:56:33.627481+0000","last_active":"2023-12-15T22:56:39.385630+0000","last_peered":"2023-12-15T22:56:39.385630+0000","last_clean":"2023-12-15T22:56:39.385630+0000","last_became_active":"2023-12-15T22:56:33.624689+0000","last_became_peered":"2023-12-15T22:56:33.624689+0000","last_unstale":"2023-12-15T22:56:39.385630+0000","last_undegraded":"2023-12-15T22:56:39.385630+0000","last_fullsized":"2023-12-15T22:56:39.385630+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-15T22:56:39.532152+0000","last_change":"2023-12-15T22:56:39.532541+0000","last_active":"2023-12-15T22:56:39.532152+0000","last_peered":"2023-12-15T22:56:39.532152+0000","last_clean":"2023-12-15T22:56:38.356579+0000","last_became_active":"2023-12-15T22:56:39.529359+0000","last_became_peered":"2023-12-15T22:56:39.529359+0000","last_unstale":"2023-12-15T22:56:39.532152+0000","last_undegraded":"2023-12-15T22:56:39.532152+0000","last_fullsized":"2023-12-15T22:56:39.532152+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-12-15T22:56:40.379247+0000","last_change":"2023-12-15T22:56:33.655797+0000","last_active":"2023-12-15T22:56:40.379247+0000","last_peered":"2023-12-15T22:56:40.379247+0000","last_clean":"2023-12-15T22:56:40.379247+0000","last_became_active":"2023-12-15T22:56:33.652885+0000","last_became_peered":"2023-12-15T22:56:33.652885+0000","last_unstale":"2023-12-15T22:56:40.379247+0000","last_undegraded":"2023-12-15T22:56:40.379247+0000","last_fullsized":"2023-12-15T22:56:40.379247+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-12-15T22:56:40.555905+0000","last_change":"2023-12-15T22:56:39.553365+0000","last_active":"2023-12-15T22:56:40.555905+0000","last_peered":"2023-12-15T22:56:40.555905+0000","last_clean":"2023-12-15T22:56:40.555905+0000","last_became_active":"2023-12-15T22:56:39.543509+0000","last_became_peered":"2023-12-15T22:56:39.543509+0000","last_unstale":"2023-12-15T22:56:40.555905+0000","last_undegraded":"2023-12-15T22:56:40.555905+0000","last_fullsized":"2023-12-15T22:56:40.555905+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-15T22:56:39.401889+0000","last_change":"2023-12-15T22:56:33.600485+0000","last_active":"2023-12-15T22:56:39.401889+0000","last_peered":"2023-12-15T22:56:39.401889+0000","last_clean":"2023-12-15T22:56:39.401889+0000","last_became_active":"2023-12-15T22:56:33.597320+0000","last_became_peered":"2023-12-15T22:56:33.597320+0000","last_unstale":"2023-12-15T22:56:39.401889+0000","last_undegraded":"2023-12-15T22:56:39.401889+0000","last_fullsized":"2023-12-15T22:56:39.401889+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-12-15T22:56:40.553130+0000","last_change":"2023-12-15T22:56:39.565715+0000","last_active":"2023-12-15T22:56:40.553130+0000","last_peered":"2023-12-15T22:56:40.553130+0000","last_clean":"2023-12-15T22:56:40.553130+0000","last_became_active":"2023-12-15T22:56:39.560980+0000","last_became_peered":"2023-12-15T22:56:39.560980+0000","last_unstale":"2023-12-15T22:56:40.553130+0000","last_undegraded":"2023-12-15T22:56:40.553130+0000","last_fullsized":"2023-12-15T22:56:40.553130+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-12-15T22:56:40.379790+0000","last_change":"2023-12-15T22:56:33.648541+0000","last_active":"2023-12-15T22:56:40.379790+0000","last_peered":"2023-12-15T22:56:40.379790+0000","last_clean":"2023-12-15T22:56:40.379790+0000","last_became_active":"2023-12-15T22:56:33.643681+0000","last_became_peered":"2023-12-15T22:56:33.643681+0000","last_unstale":"2023-12-15T22:56:40.379790+0000","last_undegraded":"2023-12-15T22:56:40.379790+0000","last_fullsized":"2023-12-15T22:56:40.379790+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-15T22:56:39.513560+0000","last_change":"2023-12-15T22:56:39.513963+0000","last_active":"2023-12-15T22:56:39.513560+0000","last_peered":"2023-12-15T22:56:39.513560+0000","last_clean":"2023-12-15T22:56:38.356579+0000","last_became_active":"2023-12-15T22:56:39.506510+0000","last_became_peered":"2023-12-15T22:56:39.506510+0000","last_unstale":"2023-12-15T22:56:39.513560+0000","last_undegraded":"2023-12-15T22:56:39.513560+0000","last_fullsized":"2023-12-15T22:56:39.513560+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-15T22:56:39.523533+0000","last_change":"2023-12-15T22:56:39.523991+0000","last_active":"2023-12-15T22:56:39.523533+0000","last_peered":"2023-12-15T22:56:39.523533+0000","last_clean":"2023-12-15T22:56:38.356579+0000","last_became_active":"2023-12-15T22:56:39.511159+0000","last_became_peered":"2023-12-15T22:56:39.511159+0000","last_unstale":"2023-12-15T22:56:39.523533+0000","last_undegraded":"2023-12-15T22:56:39.523533+0000","last_fullsized":"2023-12-15T22:56:39.523533+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":13,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-15T22:56:39.413582+0000","last_change":"2023-12-15T22:56:33.633890+0000","last_active":"2023-12-15T22:56:39.413582+0000","last_peered":"2023-12-15T22:56:39.413582+0000","last_clean":"2023-12-15T22:56:39.413582+0000","last_became_active":"2023-12-15T22:56:33.631134+0000","last_became_peered":"2023-12-15T22:56:33.631134+0000","last_unstale":"2023-12-15T22:56:39.413582+0000","last_undegraded":"2023-12-15T22:56:39.413582+0000","last_fullsized":"2023-12-15T22:56:39.413582+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-15T22:56:39.536981+0000","last_change":"2023-12-15T22:56:39.537345+0000","last_active":"2023-12-15T22:56:39.536981+0000","last_peered":"2023-12-15T22:56:39.536981+0000","last_clean":"2023-12-15T22:56:38.356579+0000","last_became_active":"2023-12-15T22:56:39.532730+0000","last_became_peered":"2023-12-15T22:56:39.532730+0000","last_unstale":"2023-12-15T22:56:39.536981+0000","last_undegraded":"2023-12-15T22:56:39.536981+0000","last_fullsized":"2023-12-15T22:56:39.536981+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-12-15T22:56:40.553923+0000","last_change":"2023-12-15T22:56:33.655736+0000","last_active":"2023-12-15T22:56:40.553923+0000","last_peered":"2023-12-15T22:56:40.553923+0000","last_clean":"2023-12-15T22:56:40.553923+0000","last_became_active":"2023-12-15T22:56:33.652917+0000","last_became_peered":"2023-12-15T22:56:33.652917+0000","last_unstale":"2023-12-15T22:56:40.553923+0000","last_undegraded":"2023-12-15T22:56:40.553923+0000","last_fullsized":"2023-12-15T22:56:40.553923+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-12-15T22:56:40.549512+0000","last_change":"2023-12-15T22:56:39.562509+0000","last_active":"2023-12-15T22:56:40.549512+0000","last_peered":"2023-12-15T22:56:40.549512+0000","last_clean":"2023-12-15T22:56:40.549512+0000","last_became_active":"2023-12-15T22:56:39.557252+0000","last_became_peered":"2023-12-15T22:56:39.557252+0000","last_unstale":"2023-12-15T22:56:40.549512+0000","last_undegraded":"2023-12-15T22:56:40.549512+0000","last_fullsized":"2023-12-15T22:56:40.549512+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":13,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-15T22:56:39.395146+0000","last_change":"2023-12-15T22:56:33.632348+0000","last_active":"2023-12-15T22:56:39.395146+0000","last_peered":"2023-12-15T22:56:39.395146+0000","last_clean":"2023-12-15T22:56:39.395146+0000","last_became_active":"2023-12-15T22:56:33.629022+0000","last_became_peered":"2023-12-15T22:56:33.629022+0000","last_unstale":"2023-12-15T22:56:39.395146+0000","last_undegraded":"2023-12-15T22:56:39.395146+0000","last_fullsized":"2023-12-15T22:56:39.395146+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-15T22:56:39.538745+0000","last_change":"2023-12-15T22:56:39.539110+0000","last_active":"2023-12-15T22:56:39.538745+0000","last_peered":"2023-12-15T22:56:39.538745+0000","last_clean":"2023-12-15T22:56:38.356579+0000","last_became_active":"2023-12-15T22:56:39.535145+0000","last_became_peered":"2023-12-15T22:56:39.535145+0000","last_unstale":"2023-12-15T22:56:39.538745+0000","last_undegraded":"2023-12-15T22:56:39.538745+0000","last_fullsized":"2023-12-15T22:56:39.538745+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-15T22:56:39.385979+0000","last_change":"2023-12-15T22:56:33.627005+0000","last_active":"2023-12-15T22:56:39.385979+0000","last_peered":"2023-12-15T22:56:39.385979+0000","last_clean":"2023-12-15T22:56:39.385979+0000","last_became_active":"2023-12-15T22:56:33.622181+0000","last_became_peered":"2023-12-15T22:56:33.622181+0000","last_unstale":"2023-12-15T22:56:39.385979+0000","last_undegraded":"2023-12-15T22:56:39.385979+0000","last_fullsized":"2023-12-15T22:56:39.385979+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705675,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26952,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344888,"statfs":{"total":96636764160,"available":96609165312,"internally_reserved":0,"allocated":131072,"data_stored":619255,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":122880,"data_stored":613269,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26884,"kb_used_data":60,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344956,"statfs":{"total":96636764160,"available":96609234944,"internally_reserved":0,"allocated":61440,"data_stored":22901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705675,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26952,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344888,"statfs":{"total":96636764160,"available":96609165312,"internally_reserved":0,"allocated":131072,"data_stored":619255,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-15T22:56:43.657 INFO:tasks.rados.rados.0.smithi071.stdout:67: expect (ObjNum 27 snap 0 seq_num 27) 2023-12-15T22:56:43.751 INFO:tasks.rados.rados.0.smithi071.stdout:69: expect (ObjNum 42 snap 0 seq_num 42) 2023-12-15T22:56:43.858 INFO:tasks.rados.rados.0.smithi071.stdout:65: finishing write tid 1 to smithi07183189-40 2023-12-15T22:56:43.858 INFO:tasks.rados.rados.0.smithi071.stdout:73: expect (ObjNum 26 snap 0 seq_num 26) 2023-12-15T22:56:43.951 INFO:tasks.rados.rados.0.smithi071.stdout:72: expect (ObjNum 7 snap 0 seq_num 7) 2023-12-15T22:56:44.035 INFO:tasks.rados.rados.0.smithi071.stdout:65: finishing write tid 2 to smithi07183189-40 2023-12-15T22:56:44.035 INFO:tasks.rados.rados.0.smithi071.stdout:65: finishing write tid 3 to smithi07183189-40 2023-12-15T22:56:44.035 INFO:tasks.rados.rados.0.smithi071.stdout:65: finishing write tid 4 to smithi07183189-40 2023-12-15T22:56:44.035 INFO:tasks.rados.rados.0.smithi071.stdout:65: finishing write tid 5 to smithi07183189-40 2023-12-15T22:56:44.035 INFO:tasks.rados.rados.0.smithi071.stdout:65: oid 40 updating version 0 to 27 2023-12-15T22:56:44.035 INFO:tasks.rados.rados.0.smithi071.stdout:65: oid 40 updating version 27 to 28 2023-12-15T22:56:44.036 INFO:tasks.rados.rados.0.smithi071.stdout:65: oid 40 updating version 28 to 29 2023-12-15T22:56:44.036 INFO:tasks.rados.rados.0.smithi071.stdout:65: oid 40 version 29 is already newer than 26 2023-12-15T22:56:44.036 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 40 v 29 (ObjNum 60 snap 1 seq_num 60) dirty exists 2023-12-15T22:56:44.036 INFO:tasks.rados.rados.0.smithi071.stdout:65: left oid 40 (ObjNum 60 snap 1 seq_num 60) 2023-12-15T22:56:44.036 INFO:tasks.rados.rados.0.smithi071.stdout:71: expect (ObjNum 49 snap 0 seq_num 49) 2023-12-15T22:56:44.381 INFO:tasks.rados.rados.0.smithi071.stdout:65: done (9 left) 2023-12-15T22:56:44.381 INFO:tasks.rados.rados.0.smithi071.stdout:66: done (8 left) 2023-12-15T22:56:44.381 INFO:tasks.rados.rados.0.smithi071.stdout:67: done (7 left) 2023-12-15T22:56:44.381 INFO:tasks.rados.rados.0.smithi071.stdout:68: done (6 left) 2023-12-15T22:56:44.381 INFO:tasks.rados.rados.0.smithi071.stdout:69: done (5 left) 2023-12-15T22:56:44.382 INFO:tasks.rados.rados.0.smithi071.stdout:70: done (4 left) 2023-12-15T22:56:44.382 INFO:tasks.rados.rados.0.smithi071.stdout:71: done (3 left) 2023-12-15T22:56:44.382 INFO:tasks.rados.rados.0.smithi071.stdout:72: done (2 left) 2023-12-15T22:56:44.382 INFO:tasks.rados.rados.0.smithi071.stdout:73: done (1 left) 2023-12-15T22:56:44.382 INFO:tasks.rados.rados.0.smithi071.stdout:74: done (0 left) 2023-12-15T22:56:44.382 INFO:tasks.rados.rados.0.smithi071.stdout:75: write oid 45 current snap is 2 2023-12-15T22:56:44.382 INFO:tasks.rados.rados.0.smithi071.stdout:75: seq_num 61 ranges {625227=721013,1952931=704669} 2023-12-15T22:56:44.388 INFO:tasks.rados.rados.0.smithi071.stdout:75: writing smithi07183189-45 from 625227 to 1346240 tid 1 2023-12-15T22:56:44.396 INFO:tasks.rados.rados.0.smithi071.stdout:75: writing smithi07183189-45 from 1952931 to 2657600 tid 2 2023-12-15T22:56:44.396 INFO:tasks.rados.rados.0.smithi071.stdout:76: read oid 16 snap -1 2023-12-15T22:56:44.396 INFO:tasks.rados.rados.0.smithi071.stdout:76: expect (ObjNum 51 snap 0 seq_num 51) 2023-12-15T22:56:44.396 INFO:tasks.rados.rados.0.smithi071.stdout:77: write oid 46 current snap is 2 2023-12-15T22:56:44.396 INFO:tasks.rados.rados.0.smithi071.stdout:77: seq_num 62 ranges {642034=796262,1895060=564680} 2023-12-15T22:56:44.404 INFO:tasks.rados.rados.0.smithi071.stdout:77: writing smithi07183189-46 from 642034 to 1438296 tid 1 2023-12-15T22:56:44.410 INFO:tasks.rados.rados.0.smithi071.stdout:77: writing smithi07183189-46 from 1895060 to 2459740 tid 2 2023-12-15T22:56:44.411 INFO:tasks.rados.rados.0.smithi071.stdout:78: read oid 26 snap 0 2023-12-15T22:56:44.411 INFO:tasks.rados.rados.0.smithi071.stdout:78: expect (ObjNum 25 snap 0 seq_num 25) 2023-12-15T22:56:44.411 INFO:tasks.rados.rados.0.smithi071.stdout:79: read oid 7 snap -1 2023-12-15T22:56:44.411 INFO:tasks.rados.rados.0.smithi071.stdout:79: expect (ObjNum 6 snap 0 seq_num 6) 2023-12-15T22:56:44.411 INFO:tasks.rados.rados.0.smithi071.stdout:80: write oid 44 current snap is 2 2023-12-15T22:56:44.411 INFO:tasks.rados.rados.0.smithi071.stdout:80: seq_num 63 ranges {658841=471511,1553542=708338,2964119=94722} 2023-12-15T22:56:44.416 INFO:tasks.rados.rados.0.smithi071.stdout:80: writing smithi07183189-44 from 658841 to 1130352 tid 1 2023-12-15T22:56:44.423 INFO:tasks.rados.rados.0.smithi071.stdout:80: writing smithi07183189-44 from 1553542 to 2261880 tid 2 2023-12-15T22:56:44.428 INFO:tasks.rados.rados.0.smithi071.stdout:80: writing smithi07183189-44 from 2964119 to 3058841 tid 3 2023-12-15T22:56:44.428 INFO:tasks.rados.rados.0.smithi071.stdout:81: read oid 30 snap -1 2023-12-15T22:56:44.428 INFO:tasks.rados.rados.0.smithi071.stdout:81: expect (ObjNum 29 snap 0 seq_num 29) 2023-12-15T22:56:44.428 INFO:tasks.rados.rados.0.smithi071.stdout:82: snap_remove snap 1 2023-12-15T22:56:44.459 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:56:44.568 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:44.566+0000 7fe9ff91e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:44.571 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:44.571+0000 7fe9ff91e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:45.395 INFO:tasks.rados.rados.0.smithi071.stdout:82: done (7 left) 2023-12-15T22:56:45.395 INFO:tasks.rados.rados.0.smithi071.stdout:83: delete oid 49 current snap is 2 2023-12-15T22:56:45.395 INFO:tasks.rados.rados.0.smithi071.stdout:76: expect (ObjNum 51 snap 0 seq_num 51) 2023-12-15T22:56:45.548 INFO:tasks.rados.rados.0.smithi071.stdout:79: expect (ObjNum 6 snap 0 seq_num 6) 2023-12-15T22:56:45.632 INFO:tasks.rados.rados.0.smithi071.stdout:78: expect (ObjNum 25 snap 0 seq_num 25) 2023-12-15T22:56:45.725 INFO:tasks.rados.rados.0.smithi071.stdout:75: finishing write tid 1 to smithi07183189-45 2023-12-15T22:56:45.725 INFO:tasks.rados.rados.0.smithi071.stdout:77: finishing write tid 1 to smithi07183189-46 2023-12-15T22:56:45.725 INFO:tasks.rados.rados.0.smithi071.stdout:81: expect (ObjNum 29 snap 0 seq_num 29) 2023-12-15T22:56:45.825 INFO:tasks.rados.rados.0.smithi071.stdout:75: finishing write tid 2 to smithi07183189-45 2023-12-15T22:56:45.825 INFO:tasks.rados.rados.0.smithi071.stdout:75: finishing write tid 3 to smithi07183189-45 2023-12-15T22:56:45.825 INFO:tasks.rados.rados.0.smithi071.stdout:75: finishing write tid 4 to smithi07183189-45 2023-12-15T22:56:45.825 INFO:tasks.rados.rados.0.smithi071.stdout:75: oid 45 updating version 0 to 32 2023-12-15T22:56:45.825 INFO:tasks.rados.rados.0.smithi071.stdout:75: oid 45 version 32 is already newer than 32 2023-12-15T22:56:45.825 INFO:tasks.rados.rados.0.smithi071.stdout:75: oid 45 updating version 32 to 33 2023-12-15T22:56:45.825 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 33 (ObjNum 61 snap 2 seq_num 61) dirty exists 2023-12-15T22:56:45.825 INFO:tasks.rados.rados.0.smithi071.stdout:75: left oid 45 (ObjNum 61 snap 2 seq_num 61) 2023-12-15T22:56:45.825 INFO:tasks.rados.rados.0.smithi071.stdout:77: finishing write tid 2 to smithi07183189-46 2023-12-15T22:56:45.825 INFO:tasks.rados.rados.0.smithi071.stdout:75: done (7 left) 2023-12-15T22:56:45.825 INFO:tasks.rados.rados.0.smithi071.stdout:76: done (6 left) 2023-12-15T22:56:45.825 INFO:tasks.rados.rados.0.smithi071.stdout:78: done (5 left) 2023-12-15T22:56:45.825 INFO:tasks.rados.rados.0.smithi071.stdout:79: done (4 left) 2023-12-15T22:56:45.825 INFO:tasks.rados.rados.0.smithi071.stdout:81: done (3 left) 2023-12-15T22:56:45.825 INFO:tasks.rados.rados.0.smithi071.stdout:83: done (2 left) 2023-12-15T22:56:45.826 INFO:tasks.rados.rados.0.smithi071.stdout:84: read oid 14 snap -1 2023-12-15T22:56:45.826 INFO:tasks.rados.rados.0.smithi071.stdout:84: expect (ObjNum 13 snap 0 seq_num 13) 2023-12-15T22:56:45.826 INFO:tasks.rados.rados.0.smithi071.stdout:85: read oid 4 snap 0 2023-12-15T22:56:45.826 INFO:tasks.rados.rados.0.smithi071.stdout:85: expect (ObjNum 3 snap 0 seq_num 3) 2023-12-15T22:56:45.826 INFO:tasks.rados.rados.0.smithi071.stdout:86: write oid 39 current snap is 2 2023-12-15T22:56:45.826 INFO:tasks.rados.rados.0.smithi071.stdout:77: finishing write tid 3 to smithi07183189-46 2023-12-15T22:56:45.826 INFO:tasks.rados.rados.0.smithi071.stdout:77: finishing write tid 4 to smithi07183189-46 2023-12-15T22:56:45.826 INFO:tasks.rados.rados.0.smithi071.stdout:77: oid 46 updating version 0 to 9 2023-12-15T22:56:45.826 INFO:tasks.rados.rados.0.smithi071.stdout:77: oid 46 updating version 9 to 10 2023-12-15T22:56:45.826 INFO:tasks.rados.rados.0.smithi071.stdout:77: oid 46 version 10 is already newer than 8 2023-12-15T22:56:45.826 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 10 (ObjNum 62 snap 2 seq_num 62) dirty exists 2023-12-15T22:56:45.826 INFO:tasks.rados.rados.0.smithi071.stdout:77: left oid 46 (ObjNum 62 snap 2 seq_num 62) 2023-12-15T22:56:45.826 INFO:tasks.rados.rados.0.smithi071.stdout:80: finishing write tid 1 to smithi07183189-44 2023-12-15T22:56:45.826 INFO:tasks.rados.rados.0.smithi071.stdout:80: finishing write tid 2 to smithi07183189-44 2023-12-15T22:56:45.826 INFO:tasks.rados.rados.0.smithi071.stdout:80: finishing write tid 3 to smithi07183189-44 2023-12-15T22:56:45.826 INFO:tasks.rados.rados.0.smithi071.stdout:80: finishing write tid 4 to smithi07183189-44 2023-12-15T22:56:45.826 INFO:tasks.rados.rados.0.smithi071.stdout:80: finishing write tid 5 to smithi07183189-44 2023-12-15T22:56:45.827 INFO:tasks.rados.rados.0.smithi071.stdout:80: oid 44 updating version 0 to 15 2023-12-15T22:56:45.827 INFO:tasks.rados.rados.0.smithi071.stdout:80: oid 44 version 15 is already newer than 13 2023-12-15T22:56:45.827 INFO:tasks.rados.rados.0.smithi071.stdout:80: oid 44 version 15 is already newer than 12 2023-12-15T22:56:45.827 INFO:tasks.rados.rados.0.smithi071.stdout:80: oid 44 version 15 is already newer than 14 2023-12-15T22:56:45.827 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 15 (ObjNum 63 snap 2 seq_num 63) dirty exists 2023-12-15T22:56:45.827 INFO:tasks.rados.rados.0.smithi071.stdout:80: left oid 44 (ObjNum 63 snap 2 seq_num 63) 2023-12-15T22:56:45.827 INFO:tasks.rados.rados.0.smithi071.stdout:86: seq_num 65 ranges {692455=622009,2070506=712007} 2023-12-15T22:56:45.832 INFO:tasks.rados.rados.0.smithi071.stdout:86: writing smithi07183189-39 from 692455 to 1314464 tid 1 2023-12-15T22:56:45.840 INFO:tasks.rados.rados.0.smithi071.stdout:86: writing smithi07183189-39 from 2070506 to 2782513 tid 2 2023-12-15T22:56:45.840 INFO:tasks.rados.rados.0.smithi071.stdout:77: done (4 left) 2023-12-15T22:56:45.840 INFO:tasks.rados.rados.0.smithi071.stdout:80: done (3 left) 2023-12-15T22:56:45.841 INFO:tasks.rados.rados.0.smithi071.stdout:87: snap_create 2023-12-15T22:56:45.865 INFO:tasks.rados.rados.0.smithi071.stdout:84: expect (ObjNum 13 snap 0 seq_num 13) 2023-12-15T22:56:45.955 INFO:tasks.rados.rados.0.smithi071.stdout:85: expect (ObjNum 3 snap 0 seq_num 3) 2023-12-15T22:56:46.034 INFO:tasks.rados.rados.0.smithi071.stdout:86: finishing write tid 1 to smithi07183189-39 2023-12-15T22:56:46.035 INFO:tasks.rados.rados.0.smithi071.stdout:86: finishing write tid 2 to smithi07183189-39 2023-12-15T22:56:46.035 INFO:tasks.rados.rados.0.smithi071.stdout:86: finishing write tid 3 to smithi07183189-39 2023-12-15T22:56:46.035 INFO:tasks.rados.rados.0.smithi071.stdout:86: finishing write tid 4 to smithi07183189-39 2023-12-15T22:56:46.035 INFO:tasks.rados.rados.0.smithi071.stdout:86: oid 39 updating version 0 to 18 2023-12-15T22:56:46.035 INFO:tasks.rados.rados.0.smithi071.stdout:86: oid 39 version 18 is already newer than 18 2023-12-15T22:56:46.035 INFO:tasks.rados.rados.0.smithi071.stdout:86: oid 39 version 18 is already newer than 16 2023-12-15T22:56:46.035 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 39 v 18 (ObjNum 65 snap 2 seq_num 65) dirty exists 2023-12-15T22:56:46.035 INFO:tasks.rados.rados.0.smithi071.stdout:86: left oid 39 (ObjNum 65 snap 2 seq_num 65) 2023-12-15T22:56:46.396 INFO:tasks.rados.rados.0.smithi071.stdout:84: done (3 left) 2023-12-15T22:56:46.396 INFO:tasks.rados.rados.0.smithi071.stdout:85: done (2 left) 2023-12-15T22:56:46.396 INFO:tasks.rados.rados.0.smithi071.stdout:86: done (1 left) 2023-12-15T22:56:46.396 INFO:tasks.rados.rados.0.smithi071.stdout:87: done (0 left) 2023-12-15T22:56:46.396 INFO:tasks.rados.rados.0.smithi071.stdout:88: write oid 6 current snap is 3 2023-12-15T22:56:46.396 INFO:tasks.rados.rados.0.smithi071.stdout:88: seq_num 66 ranges {709262=697258,2012635=572018,3046388=62874} 2023-12-15T22:56:46.403 INFO:tasks.rados.rados.0.smithi071.stdout:88: writing smithi07183189-6 from 709262 to 1406520 tid 1 2023-12-15T22:56:46.410 INFO:tasks.rados.rados.0.smithi071.stdout:88: writing smithi07183189-6 from 2012635 to 2584653 tid 2 2023-12-15T22:56:46.413 INFO:tasks.rados.rados.0.smithi071.stdout:88: writing smithi07183189-6 from 3046388 to 3109262 tid 3 2023-12-15T22:56:46.413 INFO:tasks.rados.rados.0.smithi071.stdout:89: write oid 7 current snap is 3 2023-12-15T22:56:46.413 INFO:tasks.rados.rados.0.smithi071.stdout:89: seq_num 67 ranges {726069=772507,2071117=715676} 2023-12-15T22:56:46.421 INFO:tasks.rados.rados.0.smithi071.stdout:89: writing smithi07183189-7 from 726069 to 1498576 tid 1 2023-12-15T22:56:46.429 INFO:tasks.rados.rados.0.smithi071.stdout:89: writing smithi07183189-7 from 2071117 to 2786793 tid 2 2023-12-15T22:56:46.429 INFO:tasks.rados.rados.0.smithi071.stdout:90: write oid 22 current snap is 3 2023-12-15T22:56:46.429 INFO:tasks.rados.rados.0.smithi071.stdout:90: seq_num 68 ranges {742876=447756,1729599=575687,3101234=41642} 2023-12-15T22:56:46.436 INFO:tasks.rados.rados.0.smithi071.stdout:90: writing smithi07183189-22 from 742876 to 1190632 tid 1 2023-12-15T22:56:46.443 INFO:tasks.rados.rados.0.smithi071.stdout:90: writing smithi07183189-22 from 1729599 to 2305286 tid 2 2023-12-15T22:56:46.448 INFO:tasks.rados.rados.0.smithi071.stdout:90: writing smithi07183189-22 from 3101234 to 3142876 tid 3 2023-12-15T22:56:46.448 INFO:tasks.rados.rados.0.smithi071.stdout:91: delete oid 14 current snap is 3 2023-12-15T22:56:46.463 INFO:tasks.rados.rados.0.smithi071.stdout:89: finishing write tid 1 to smithi07183189-7 2023-12-15T22:56:46.485 INFO:tasks.rados.rados.0.smithi071.stdout:88: finishing write tid 1 to smithi07183189-6 2023-12-15T22:56:46.506 INFO:tasks.rados.rados.0.smithi071.stdout:89: finishing write tid 2 to smithi07183189-7 2023-12-15T22:56:46.527 INFO:tasks.rados.rados.0.smithi071.stdout:90: finishing write tid 1 to smithi07183189-22 2023-12-15T22:56:46.537 INFO:tasks.rados.rados.0.smithi071.stdout:88: finishing write tid 2 to smithi07183189-6 2023-12-15T22:56:46.546 INFO:tasks.rados.rados.0.smithi071.stdout:89: finishing write tid 3 to smithi07183189-7 2023-12-15T22:56:46.548 INFO:tasks.rados.rados.0.smithi071.stdout:89: finishing write tid 4 to smithi07183189-7 2023-12-15T22:56:46.548 INFO:tasks.rados.rados.0.smithi071.stdout:89: oid 7 updating version 0 to 23 2023-12-15T22:56:46.548 INFO:tasks.rados.rados.0.smithi071.stdout:89: oid 7 updating version 23 to 25 2023-12-15T22:56:46.548 INFO:tasks.rados.rados.0.smithi071.stdout:89: oid 7 version 25 is already newer than 24 2023-12-15T22:56:46.548 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 25 (ObjNum 67 snap 3 seq_num 67) dirty exists 2023-12-15T22:56:46.548 INFO:tasks.rados.rados.0.smithi071.stdout:89: left oid 7 (ObjNum 67 snap 3 seq_num 67) 2023-12-15T22:56:46.553 INFO:tasks.rados.rados.0.smithi071.stdout:88: finishing write tid 3 to smithi07183189-6 2023-12-15T22:56:46.566 INFO:tasks.rados.rados.0.smithi071.stdout:88: finishing write tid 4 to smithi07183189-6 2023-12-15T22:56:46.570 INFO:tasks.rados.rados.0.smithi071.stdout:88: finishing write tid 5 to smithi07183189-6 2023-12-15T22:56:46.570 INFO:tasks.rados.rados.0.smithi071.stdout:88: oid 6 updating version 0 to 18 2023-12-15T22:56:46.570 INFO:tasks.rados.rados.0.smithi071.stdout:88: oid 6 version 18 is already newer than 15 2023-12-15T22:56:46.570 INFO:tasks.rados.rados.0.smithi071.stdout:88: oid 6 version 18 is already newer than 17 2023-12-15T22:56:46.570 INFO:tasks.rados.rados.0.smithi071.stdout:88: oid 6 version 18 is already newer than 16 2023-12-15T22:56:46.570 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 6 v 18 (ObjNum 66 snap 3 seq_num 66) dirty exists 2023-12-15T22:56:46.570 INFO:tasks.rados.rados.0.smithi071.stdout:88: left oid 6 (ObjNum 66 snap 3 seq_num 66) 2023-12-15T22:56:46.584 INFO:tasks.rados.rados.0.smithi071.stdout:88: done (3 left) 2023-12-15T22:56:46.584 INFO:tasks.rados.rados.0.smithi071.stdout:89: done (2 left) 2023-12-15T22:56:46.585 INFO:tasks.rados.rados.0.smithi071.stdout:91: done (1 left) 2023-12-15T22:56:46.585 INFO:tasks.rados.rados.0.smithi071.stdout:92: write (excl) oid 30 current snap is 3 2023-12-15T22:56:46.585 INFO:tasks.rados.rados.0.smithi071.stdout:92: seq_num 70 ranges {776490=714607,2246563=579356} 2023-12-15T22:56:46.592 INFO:tasks.rados.rados.0.smithi071.stdout:92: writing smithi07183189-30 from 776490 to 1491097 tid 1 2023-12-15T22:56:46.600 INFO:tasks.rados.rados.0.smithi071.stdout:92: writing smithi07183189-30 from 2246563 to 2825919 tid 2 2023-12-15T22:56:46.600 INFO:tasks.rados.rados.0.smithi071.stdout:93: write oid 29 current snap is 3 2023-12-15T22:56:46.600 INFO:tasks.rados.rados.0.smithi071.stdout:93: seq_num 71 ranges {793297=789856,2305045=723014} 2023-12-15T22:56:46.609 INFO:tasks.rados.rados.0.smithi071.stdout:93: writing smithi07183189-29 from 793297 to 1583153 tid 1 2023-12-15T22:56:46.618 INFO:tasks.rados.rados.0.smithi071.stdout:93: writing smithi07183189-29 from 2305045 to 3028059 tid 2 2023-12-15T22:56:46.618 INFO:tasks.rados.rados.0.smithi071.stdout:94: read oid 31 snap -1 2023-12-15T22:56:46.618 INFO:tasks.rados.rados.0.smithi071.stdout:94: expect deleted 2023-12-15T22:56:46.618 INFO:tasks.rados.rados.0.smithi071.stdout:95: read oid 37 snap -1 2023-12-15T22:56:46.618 INFO:tasks.rados.rados.0.smithi071.stdout:95: expect (ObjNum 36 snap 0 seq_num 36) 2023-12-15T22:56:46.618 INFO:tasks.rados.rados.0.smithi071.stdout:96: delete oid 50 current snap is 3 2023-12-15T22:56:46.618 INFO:tasks.rados.rados.0.smithi071.stdout:90: finishing write tid 2 to smithi07183189-22 2023-12-15T22:56:46.618 INFO:tasks.rados.rados.0.smithi071.stdout:90: finishing write tid 3 to smithi07183189-22 2023-12-15T22:56:46.620 INFO:tasks.rados.rados.0.smithi071.stdout:90: finishing write tid 4 to smithi07183189-22 2023-12-15T22:56:46.621 INFO:tasks.rados.rados.0.smithi071.stdout:90: finishing write tid 5 to smithi07183189-22 2023-12-15T22:56:46.621 INFO:tasks.rados.rados.0.smithi071.stdout:90: oid 22 updating version 0 to 22 2023-12-15T22:56:46.621 INFO:tasks.rados.rados.0.smithi071.stdout:90: oid 22 version 22 is already newer than 21 2023-12-15T22:56:46.621 INFO:tasks.rados.rados.0.smithi071.stdout:90: oid 22 version 22 is already newer than 20 2023-12-15T22:56:46.621 INFO:tasks.rados.rados.0.smithi071.stdout:90: oid 22 updating version 22 to 23 2023-12-15T22:56:46.621 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 23 (ObjNum 68 snap 3 seq_num 68) dirty exists 2023-12-15T22:56:46.621 INFO:tasks.rados.rados.0.smithi071.stdout:90: left oid 22 (ObjNum 68 snap 3 seq_num 68) 2023-12-15T22:56:46.625 INFO:tasks.rados.rados.0.smithi071.stdout:92: finishing write tid 1 to smithi07183189-30 2023-12-15T22:56:46.638 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:56:46.646 INFO:tasks.rados.rados.0.smithi071.stdout:92: finishing write tid 2 to smithi07183189-30 2023-12-15T22:56:46.648 INFO:tasks.rados.rados.0.smithi071.stdout:93: finishing write tid 1 to smithi07183189-29 2023-12-15T22:56:46.654 INFO:tasks.rados.rados.0.smithi071.stdout:92: finishing write tid 3 to smithi07183189-30 2023-12-15T22:56:46.654 INFO:tasks.rados.rados.0.smithi071.stdout:92: finishing write tid 4 to smithi07183189-30 2023-12-15T22:56:46.654 INFO:tasks.rados.rados.0.smithi071.stdout:92: oid 30 updating version 0 to 9 2023-12-15T22:56:46.654 INFO:tasks.rados.rados.0.smithi071.stdout:92: oid 30 version 9 is already newer than 8 2023-12-15T22:56:46.654 INFO:tasks.rados.rados.0.smithi071.stdout:92: oid 30 updating version 9 to 10 2023-12-15T22:56:46.655 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 10 (ObjNum 70 snap 3 seq_num 70) dirty exists 2023-12-15T22:56:46.655 INFO:tasks.rados.rados.0.smithi071.stdout:92: left oid 30 (ObjNum 70 snap 3 seq_num 70) 2023-12-15T22:56:46.668 INFO:tasks.rados.rados.0.smithi071.stdout:93: finishing write tid 2 to smithi07183189-29 2023-12-15T22:56:46.669 INFO:tasks.rados.rados.0.smithi071.stdout:93: finishing write tid 3 to smithi07183189-29 2023-12-15T22:56:46.673 INFO:tasks.rados.rados.0.smithi071.stdout:93: finishing write tid 4 to smithi07183189-29 2023-12-15T22:56:46.673 INFO:tasks.rados.rados.0.smithi071.stdout:93: oid 29 updating version 0 to 39 2023-12-15T22:56:46.673 INFO:tasks.rados.rados.0.smithi071.stdout:93: oid 29 version 39 is already newer than 37 2023-12-15T22:56:46.673 INFO:tasks.rados.rados.0.smithi071.stdout:93: oid 29 version 39 is already newer than 39 2023-12-15T22:56:46.673 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 39 (ObjNum 71 snap 3 seq_num 71) dirty exists 2023-12-15T22:56:46.673 INFO:tasks.rados.rados.0.smithi071.stdout:93: left oid 29 (ObjNum 71 snap 3 seq_num 71) 2023-12-15T22:56:46.685 INFO:tasks.rados.rados.0.smithi071.stdout:95: expect (ObjNum 36 snap 0 seq_num 36) 2023-12-15T22:56:46.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:46.743+0000 7fc115e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:46.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:46.748+0000 7fc115e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:46.793 INFO:tasks.rados.rados.0.smithi071.stdout:90: done (5 left) 2023-12-15T22:56:46.793 INFO:tasks.rados.rados.0.smithi071.stdout:92: done (4 left) 2023-12-15T22:56:46.793 INFO:tasks.rados.rados.0.smithi071.stdout:93: done (3 left) 2023-12-15T22:56:46.793 INFO:tasks.rados.rados.0.smithi071.stdout:94: done (2 left) 2023-12-15T22:56:46.793 INFO:tasks.rados.rados.0.smithi071.stdout:95: done (1 left) 2023-12-15T22:56:46.793 INFO:tasks.rados.rados.0.smithi071.stdout:96: done (0 left) 2023-12-15T22:56:46.794 INFO:tasks.rados.rados.0.smithi071.stdout:97: snap_remove snap 0 2023-12-15T22:56:47.405 INFO:tasks.rados.rados.0.smithi071.stdout:97: done (0 left) 2023-12-15T22:56:47.405 INFO:tasks.rados.rados.0.smithi071.stdout:98: write (excl) oid 43 current snap is 3 2023-12-15T22:56:47.405 INFO:tasks.rados.rados.0.smithi071.stdout:98: seq_num 73 ranges {426911=540354,1622009=726683,3121996=104915} 2023-12-15T22:56:47.410 INFO:tasks.rados.rados.0.smithi071.stdout:98: writing smithi07183189-43 from 426911 to 967265 tid 1 2023-12-15T22:56:47.418 INFO:tasks.rados.rados.0.smithi071.stdout:98: writing smithi07183189-43 from 1622009 to 2348692 tid 2 2023-12-15T22:56:47.422 INFO:tasks.rados.rados.0.smithi071.stdout:98: writing smithi07183189-43 from 3121996 to 3226911 tid 3 2023-12-15T22:56:47.422 INFO:tasks.rados.rados.0.smithi071.stdout:99: snap_remove snap 2 2023-12-15T22:56:48.409 INFO:tasks.rados.rados.0.smithi071.stdout:99: done (1 left) 2023-12-15T22:56:48.409 INFO:tasks.rados.rados.0.smithi071.stdout:100: write oid 27 current snap is 3 2023-12-15T22:56:48.409 INFO:tasks.rados.rados.0.smithi071.stdout:98: finishing write tid 1 to smithi07183189-43 2023-12-15T22:56:48.409 INFO:tasks.rados.rados.0.smithi071.stdout:100: seq_num 74 ranges {443718=615603,1564138=470341,2633066=610652} 2023-12-15T22:56:48.415 INFO:tasks.rados.rados.0.smithi071.stdout:100: writing smithi07183189-27 from 443718 to 1059321 tid 1 2023-12-15T22:56:48.420 INFO:tasks.rados.rados.0.smithi071.stdout:100: writing smithi07183189-27 from 1564138 to 2034479 tid 2 2023-12-15T22:56:48.428 INFO:tasks.rados.rados.0.smithi071.stdout:100: writing smithi07183189-27 from 2633066 to 3243718 tid 3 2023-12-15T22:56:48.428 INFO:tasks.rados.rados.0.smithi071.stdout:101: read oid 34 snap -1 2023-12-15T22:56:48.428 INFO:tasks.rados.rados.0.smithi071.stdout:101: expect (ObjNum 33 snap 0 seq_num 33) 2023-12-15T22:56:48.428 INFO:tasks.rados.rados.0.smithi071.stdout:102: read oid 4 snap -1 2023-12-15T22:56:48.428 INFO:tasks.rados.rados.0.smithi071.stdout:102: expect (ObjNum 3 snap 0 seq_num 3) 2023-12-15T22:56:48.428 INFO:tasks.rados.rados.0.smithi071.stdout:103: read oid 19 snap -1 2023-12-15T22:56:48.428 INFO:tasks.rados.rados.0.smithi071.stdout:103: expect (ObjNum 18 snap 0 seq_num 18) 2023-12-15T22:56:48.428 INFO:tasks.rados.rados.0.smithi071.stdout:104: read oid 35 snap -1 2023-12-15T22:56:48.429 INFO:tasks.rados.rados.0.smithi071.stdout:104: expect (ObjNum 34 snap 0 seq_num 34) 2023-12-15T22:56:48.429 INFO:tasks.rados.rados.0.smithi071.stdout:105: read oid 45 snap -1 2023-12-15T22:56:48.429 INFO:tasks.rados.rados.0.smithi071.stdout:105: expect (ObjNum 61 snap 2 seq_num 61) 2023-12-15T22:56:48.429 INFO:tasks.rados.rados.0.smithi071.stdout:98: finishing write tid 2 to smithi07183189-43 2023-12-15T22:56:48.429 INFO:tasks.rados.rados.0.smithi071.stdout:98: finishing write tid 3 to smithi07183189-43 2023-12-15T22:56:48.429 INFO:tasks.rados.rados.0.smithi071.stdout:98: finishing write tid 4 to smithi07183189-43 2023-12-15T22:56:48.429 INFO:tasks.rados.rados.0.smithi071.stdout:98: finishing write tid 5 to smithi07183189-43 2023-12-15T22:56:48.429 INFO:tasks.rados.rados.0.smithi071.stdout:98: oid 43 updating version 0 to 16 2023-12-15T22:56:48.429 INFO:tasks.rados.rados.0.smithi071.stdout:98: oid 43 version 16 is already newer than 14 2023-12-15T22:56:48.429 INFO:tasks.rados.rados.0.smithi071.stdout:98: oid 43 updating version 16 to 17 2023-12-15T22:56:48.429 INFO:tasks.rados.rados.0.smithi071.stdout:98: oid 43 version 17 is already newer than 15 2023-12-15T22:56:48.429 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 17 (ObjNum 73 snap 3 seq_num 73) dirty exists 2023-12-15T22:56:48.429 INFO:tasks.rados.rados.0.smithi071.stdout:98: left oid 43 (ObjNum 73 snap 3 seq_num 73) 2023-12-15T22:56:48.429 INFO:tasks.rados.rados.0.smithi071.stdout:106: read oid 26 snap -1 2023-12-15T22:56:48.430 INFO:tasks.rados.rados.0.smithi071.stdout:106: expect (ObjNum 25 snap 0 seq_num 25) 2023-12-15T22:56:48.430 INFO:tasks.rados.rados.0.smithi071.stdout:98: done (7 left) 2023-12-15T22:56:48.430 INFO:tasks.rados.rados.0.smithi071.stdout:107: delete oid 48 current snap is 3 2023-12-15T22:56:48.489 INFO:tasks.rados.rados.0.smithi071.stdout:106: expect (ObjNum 25 snap 0 seq_num 25) 2023-12-15T22:56:48.574 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T22:56:48.574 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T22:56:48.574 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-12-15T22:56:48.574 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-12-15T22:56:48.634 INFO:tasks.rados.rados.0.smithi071.stdout:101: expect (ObjNum 33 snap 0 seq_num 33) 2023-12-15T22:56:48.686 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:48.682+0000 7f3ec059e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:48.689 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:48.689+0000 7f3ec059e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:48.703 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:56:48.703 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:56:48.704 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:56:48.704 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:56:48.704 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:56:48.704 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:56:48.704 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:56:48.704 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:56:48.704 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:56:48.704 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:56:48.704 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:56:48.704 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:56:48.706 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:56:48.706 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:56:48.706 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-12-15T22:56:48.743 INFO:tasks.rados.rados.0.smithi071.stdout:101: done (7 left) 2023-12-15T22:56:48.743 INFO:tasks.rados.rados.0.smithi071.stdout:106: done (6 left) 2023-12-15T22:56:48.743 INFO:tasks.rados.rados.0.smithi071.stdout:107: done (5 left) 2023-12-15T22:56:48.743 INFO:tasks.rados.rados.0.smithi071.stdout:108: snap_create 2023-12-15T22:56:48.743 INFO:tasks.rados.rados.0.smithi071.stdout:100: finishing write tid 1 to smithi07183189-27 2023-12-15T22:56:48.743 INFO:tasks.rados.rados.0.smithi071.stdout:102: expect (ObjNum 3 snap 0 seq_num 3) 2023-12-15T22:56:48.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:48.807+0000 7fe704b1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:48.813 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:56:48.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:48.814+0000 7fe704b1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:48.827 INFO:tasks.rados.rados.0.smithi071.stdout:103: expect (ObjNum 18 snap 0 seq_num 18) 2023-12-15T22:56:48.830 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:56:48.830 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:56:48.830 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:56:48.830 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:56:48.830 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:56:48.830 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:56:48.830 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:56:48.830 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:56:48.830 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:56:48.830 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:56:48.830 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:56:48.830 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:56:48.832 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:56:48.832 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:56:48.832 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-12-15T22:56:48.918 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:48.917+0000 7f131391e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:48.921 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:48.921+0000 7f131391e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:48.921 INFO:tasks.rados.rados.0.smithi071.stdout:105: expect (ObjNum 61 snap 2 seq_num 61) 2023-12-15T22:56:48.992 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:48.987+0000 7fb97771e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:48.995 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:48.995+0000 7fb97771e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:49.012 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:56:49.012 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:56:49.012 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:56:49.012 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:56:49.012 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:56:49.012 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:56:49.012 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:56:49.012 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:56:49.012 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:56:49.012 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:56:49.012 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:56:49.012 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:56:49.014 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:56:49.014 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:56:49.015 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-12-15T22:56:49.100 INFO:tasks.rados.rados.0.smithi071.stdout:104: expect (ObjNum 34 snap 0 seq_num 34) 2023-12-15T22:56:49.160 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:49.156+0000 7f5de5c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:49.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:49.163+0000 7f5de5c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:49.175 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:56:49.175 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:56:49.176 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:56:49.176 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:56:49.176 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:56:49.176 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:56:49.176 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:56:49.176 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:56:49.176 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:56:49.176 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:56:49.176 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:56:49.176 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:56:49.178 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:56:49.179 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:56:49.204 INFO:tasks.rados.rados.0.smithi071.stdout:100: finishing write tid 2 to smithi07183189-27 2023-12-15T22:56:49.204 INFO:tasks.rados.rados.0.smithi071.stdout:100: finishing write tid 3 to smithi07183189-27 2023-12-15T22:56:49.204 INFO:tasks.rados.rados.0.smithi071.stdout:100: finishing write tid 4 to smithi07183189-27 2023-12-15T22:56:49.204 INFO:tasks.rados.rados.0.smithi071.stdout:100: finishing write tid 5 to smithi07183189-27 2023-12-15T22:56:49.204 INFO:tasks.rados.rados.0.smithi071.stdout:100: oid 27 updating version 0 to 30 2023-12-15T22:56:49.204 INFO:tasks.rados.rados.0.smithi071.stdout:100: oid 27 updating version 30 to 31 2023-12-15T22:56:49.204 INFO:tasks.rados.rados.0.smithi071.stdout:100: oid 27 updating version 31 to 35 2023-12-15T22:56:49.204 INFO:tasks.rados.rados.0.smithi071.stdout:100: oid 27 version 35 is already newer than 35 2023-12-15T22:56:49.204 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 35 (ObjNum 74 snap 3 seq_num 74) dirty exists 2023-12-15T22:56:49.204 INFO:tasks.rados.rados.0.smithi071.stdout:100: left oid 27 (ObjNum 74 snap 3 seq_num 74) 2023-12-15T22:56:49.417 INFO:tasks.rados.rados.0.smithi071.stdout:100: done (5 left) 2023-12-15T22:56:49.417 INFO:tasks.rados.rados.0.smithi071.stdout:102: done (4 left) 2023-12-15T22:56:49.417 INFO:tasks.rados.rados.0.smithi071.stdout:103: done (3 left) 2023-12-15T22:56:49.417 INFO:tasks.rados.rados.0.smithi071.stdout:104: done (2 left) 2023-12-15T22:56:49.417 INFO:tasks.rados.rados.0.smithi071.stdout:105: done (1 left) 2023-12-15T22:56:49.417 INFO:tasks.rados.rados.0.smithi071.stdout:108: done (0 left) 2023-12-15T22:56:49.417 INFO:tasks.rados.rados.0.smithi071.stdout:109: delete oid 42 current snap is 4 2023-12-15T22:56:49.454 INFO:tasks.rados.rados.0.smithi071.stdout:109: done (0 left) 2023-12-15T22:56:49.454 INFO:tasks.rados.rados.0.smithi071.stdout:110: read oid 48 snap 3 2023-12-15T22:56:49.454 INFO:tasks.rados.rados.0.smithi071.stdout:110: expect deleted 2023-12-15T22:56:49.454 INFO:tasks.rados.rados.0.smithi071.stdout:111: snap_create 2023-12-15T22:56:50.420 INFO:tasks.rados.rados.0.smithi071.stdout:110: done (1 left) 2023-12-15T22:56:50.421 INFO:tasks.rados.rados.0.smithi071.stdout:111: done (0 left) 2023-12-15T22:56:50.421 INFO:tasks.rados.rados.0.smithi071.stdout:112: snap_create 2023-12-15T22:56:51.014 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:56:51.130 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:51.129+0000 7f554919e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:51.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:51.133+0000 7f554919e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:51.424 INFO:tasks.rados.rados.0.smithi071.stdout:112: done (0 left) 2023-12-15T22:56:51.424 INFO:tasks.rados.rados.0.smithi071.stdout:113: read oid 35 snap -1 2023-12-15T22:56:51.424 INFO:tasks.rados.rados.0.smithi071.stdout:113: expect (ObjNum 34 snap 0 seq_num 34) 2023-12-15T22:56:51.424 INFO:tasks.rados.rados.0.smithi071.stdout:114: read oid 9 snap -1 2023-12-15T22:56:51.424 INFO:tasks.rados.rados.0.smithi071.stdout:114: expect (ObjNum 53 snap 0 seq_num 53) 2023-12-15T22:56:51.424 INFO:tasks.rados.rados.0.smithi071.stdout:115: delete oid 26 current snap is 6 2023-12-15T22:56:51.454 INFO:tasks.rados.rados.0.smithi071.stdout:113: expect (ObjNum 34 snap 0 seq_num 34) 2023-12-15T22:56:51.558 INFO:tasks.rados.rados.0.smithi071.stdout:114: expect (ObjNum 53 snap 0 seq_num 53) 2023-12-15T22:56:51.709 INFO:tasks.rados.rados.0.smithi071.stdout:113: done (2 left) 2023-12-15T22:56:51.709 INFO:tasks.rados.rados.0.smithi071.stdout:114: done (1 left) 2023-12-15T22:56:51.709 INFO:tasks.rados.rados.0.smithi071.stdout:115: done (0 left) 2023-12-15T22:56:51.709 INFO:tasks.rados.rados.0.smithi071.stdout:116: write oid 28 current snap is 6 2023-12-15T22:56:51.709 INFO:tasks.rados.rados.0.smithi071.stdout:116: seq_num 78 ranges {510946=632952,1798066=477679,2742758=457217} 2023-12-15T22:56:51.715 INFO:tasks.rados.rados.0.smithi071.stdout:116: writing smithi07183189-28 from 510946 to 1143898 tid 1 2023-12-15T22:56:51.721 INFO:tasks.rados.rados.0.smithi071.stdout:116: writing smithi07183189-28 from 1798066 to 2275745 tid 2 2023-12-15T22:56:51.726 INFO:tasks.rados.rados.0.smithi071.stdout:116: writing smithi07183189-28 from 2742758 to 3199975 tid 3 2023-12-15T22:56:51.726 INFO:tasks.rados.rados.0.smithi071.stdout:117: write oid 50 current snap is 6 2023-12-15T22:56:51.726 INFO:tasks.rados.rados.0.smithi071.stdout:117: seq_num 79 ranges {527753=708201,1856548=737690,3286534=41219} 2023-12-15T22:56:51.733 INFO:tasks.rados.rados.0.smithi071.stdout:117: writing smithi07183189-50 from 527753 to 1235954 tid 1 2023-12-15T22:56:51.740 INFO:tasks.rados.rados.0.smithi071.stdout:117: writing smithi07183189-50 from 1856548 to 2594238 tid 2 2023-12-15T22:56:51.745 INFO:tasks.rados.rados.0.smithi071.stdout:117: writing smithi07183189-50 from 3286534 to 3327753 tid 3 2023-12-15T22:56:51.745 INFO:tasks.rados.rados.0.smithi071.stdout:118: write oid 49 current snap is 6 2023-12-15T22:56:51.745 INFO:tasks.rados.rados.0.smithi071.stdout:118: seq_num 80 ranges {544560=783450,1915030=481348,2797604=479761} 2023-12-15T22:56:51.752 INFO:tasks.rados.rados.0.smithi071.stdout:118: writing smithi07183189-49 from 544560 to 1328010 tid 1 2023-12-15T22:56:51.758 INFO:tasks.rados.rados.0.smithi071.stdout:118: writing smithi07183189-49 from 1915030 to 2396378 tid 2 2023-12-15T22:56:51.763 INFO:tasks.rados.rados.0.smithi071.stdout:118: writing smithi07183189-49 from 2797604 to 3277365 tid 3 2023-12-15T22:56:51.763 INFO:tasks.rados.rados.0.smithi071.stdout:119: write (excl) oid 26 current snap is 6 2023-12-15T22:56:51.763 INFO:tasks.rados.rados.0.smithi071.stdout:117: finishing write tid 1 to smithi07183189-50 2023-12-15T22:56:51.763 INFO:tasks.rados.rados.0.smithi071.stdout:119: seq_num 81 ranges {561367=458699,1573512=741359,2941380=419987} 2023-12-15T22:56:51.769 INFO:tasks.rados.rados.0.smithi071.stdout:119: writing smithi07183189-26 from 561367 to 1020066 tid 1 2023-12-15T22:56:51.776 INFO:tasks.rados.rados.0.smithi071.stdout:119: writing smithi07183189-26 from 1573512 to 2314871 tid 2 2023-12-15T22:56:51.782 INFO:tasks.rados.rados.0.smithi071.stdout:119: writing smithi07183189-26 from 2941380 to 3361367 tid 3 2023-12-15T22:56:51.783 INFO:tasks.rados.rados.0.smithi071.stdout:120: write (excl) oid 44 current snap is 6 2023-12-15T22:56:51.783 INFO:tasks.rados.rados.0.smithi071.stdout:116: finishing write tid 1 to smithi07183189-28 2023-12-15T22:56:51.783 INFO:tasks.rados.rados.0.smithi071.stdout:120: seq_num 82 ranges {578174=533948,1515641=485017,2736097=502305} 2023-12-15T22:56:51.789 INFO:tasks.rados.rados.0.smithi071.stdout:120: writing smithi07183189-44 from 578174 to 1112122 tid 1 2023-12-15T22:56:51.794 INFO:tasks.rados.rados.0.smithi071.stdout:120: writing smithi07183189-44 from 1515641 to 2000658 tid 2 2023-12-15T22:56:51.802 INFO:tasks.rados.rados.0.smithi071.stdout:120: writing smithi07183189-44 from 2736097 to 3238402 tid 3 2023-12-15T22:56:51.802 INFO:tasks.rados.rados.0.smithi071.stdout:121: read oid 22 snap -1 2023-12-15T22:56:51.802 INFO:tasks.rados.rados.0.smithi071.stdout:121: expect (ObjNum 68 snap 3 seq_num 68) 2023-12-15T22:56:51.802 INFO:tasks.rados.rados.0.smithi071.stdout:122: snap_remove snap 5 2023-12-15T22:56:52.427 INFO:tasks.rados.rados.0.smithi071.stdout:122: done (6 left) 2023-12-15T22:56:52.427 INFO:tasks.rados.rados.0.smithi071.stdout:123: read oid 37 snap -1 2023-12-15T22:56:52.427 INFO:tasks.rados.rados.0.smithi071.stdout:123: expect (ObjNum 36 snap 0 seq_num 36) 2023-12-15T22:56:52.427 INFO:tasks.rados.rados.0.smithi071.stdout:124: snap_remove snap 4 2023-12-15T22:56:53.205 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:56:53.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:53.309+0000 7f904df9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:53.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:53.313+0000 7f904df9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:53.430 INFO:tasks.rados.rados.0.smithi071.stdout:124: done (7 left) 2023-12-15T22:56:53.430 INFO:tasks.rados.rados.0.smithi071.stdout:125: snap_remove snap 3 2023-12-15T22:56:54.180 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T22:56:54.180 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T22:56:54.180 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T22:56:54.292 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:54.289+0000 7f32ac59e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:54.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:54.296+0000 7f32ac59e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:54.432 INFO:tasks.rados.rados.0.smithi071.stdout:125: done (7 left) 2023-12-15T22:56:54.432 INFO:tasks.rados.rados.0.smithi071.stdout:126: write oid 36 current snap is 6 2023-12-15T22:56:54.432 INFO:tasks.rados.rados.0.smithi071.stdout:117: finishing write tid 2 to smithi07183189-50 2023-12-15T22:56:54.432 INFO:tasks.rados.rados.0.smithi071.stdout:126: seq_num 83 ranges {594981=609197,1974123=745028,3279873=115108} 2023-12-15T22:56:54.438 INFO:tasks.rados.rados.0.smithi071.stdout:126: writing smithi07183189-36 from 594981 to 1204178 tid 1 2023-12-15T22:56:54.447 INFO:tasks.rados.rados.0.smithi071.stdout:126: writing smithi07183189-36 from 1974123 to 2719151 tid 2 2023-12-15T22:56:54.451 INFO:tasks.rados.rados.0.smithi071.stdout:126: writing smithi07183189-36 from 3279873 to 3394981 tid 3 2023-12-15T22:56:54.451 INFO:tasks.rados.rados.0.smithi071.stdout:127: write (excl) oid 35 current snap is 6 2023-12-15T22:56:54.451 INFO:tasks.rados.rados.0.smithi071.stdout:118: finishing write tid 1 to smithi07183189-49 2023-12-15T22:56:54.451 INFO:tasks.rados.rados.0.smithi071.stdout:127: seq_num 84 ranges {611788=400799,1748958=488686,2907296=504492} 2023-12-15T22:56:54.457 INFO:tasks.rados.rados.0.smithi071.stdout:127: writing smithi07183189-35 from 611788 to 1012587 tid 1 2023-12-15T22:56:54.464 INFO:tasks.rados.rados.0.smithi071.stdout:127: writing smithi07183189-35 from 1748958 to 2237644 tid 2 2023-12-15T22:56:54.470 INFO:tasks.rados.rados.0.smithi071.stdout:127: writing smithi07183189-35 from 2907296 to 3411788 tid 3 2023-12-15T22:56:54.470 INFO:tasks.rados.rados.0.smithi071.stdout:128: write oid 2 current snap is 6 2023-12-15T22:56:54.471 INFO:tasks.rados.rados.0.smithi071.stdout:128: seq_num 85 ranges {628595=476048,1807440=632344,2934719=452474} 2023-12-15T22:56:54.476 INFO:tasks.rados.rados.0.smithi071.stdout:128: writing smithi07183189-2 from 628595 to 1104643 tid 1 2023-12-15T22:56:54.484 INFO:tasks.rados.rados.0.smithi071.stdout:128: writing smithi07183189-2 from 1807440 to 2439784 tid 2 2023-12-15T22:56:54.486 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:56:54.487 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-12-15T22:56:54.491 INFO:tasks.rados.rados.0.smithi071.stdout:128: writing smithi07183189-2 from 2934719 to 3387193 tid 3 2023-12-15T22:56:54.491 INFO:tasks.rados.rados.0.smithi071.stdout:129: read oid 9 snap -1 2023-12-15T22:56:54.491 INFO:tasks.rados.rados.0.smithi071.stdout:129: expect (ObjNum 53 snap 0 seq_num 53) 2023-12-15T22:56:54.491 INFO:tasks.rados.rados.0.smithi071.stdout:116: finishing write tid 2 to smithi07183189-28 2023-12-15T22:56:54.491 INFO:tasks.rados.rados.0.smithi071.stdout:119: finishing write tid 1 to smithi07183189-26 2023-12-15T22:56:54.491 INFO:tasks.rados.rados.0.smithi071.stdout:117: finishing write tid 3 to smithi07183189-50 2023-12-15T22:56:54.491 INFO:tasks.rados.rados.0.smithi071.stdout:118: finishing write tid 2 to smithi07183189-49 2023-12-15T22:56:54.491 INFO:tasks.rados.rados.0.smithi071.stdout:119: finishing write tid 2 to smithi07183189-26 2023-12-15T22:56:54.491 INFO:tasks.rados.rados.0.smithi071.stdout:120: finishing write tid 1 to smithi07183189-44 2023-12-15T22:56:54.491 INFO:tasks.rados.rados.0.smithi071.stdout:117: finishing write tid 4 to smithi07183189-50 2023-12-15T22:56:54.491 INFO:tasks.rados.rados.0.smithi071.stdout:116: finishing write tid 3 to smithi07183189-28 2023-12-15T22:56:54.491 INFO:tasks.rados.rados.0.smithi071.stdout:117: finishing write tid 5 to smithi07183189-50 2023-12-15T22:56:54.492 INFO:tasks.rados.rados.0.smithi071.stdout:117: oid 50 updating version 0 to 54 2023-12-15T22:56:54.492 INFO:tasks.rados.rados.0.smithi071.stdout:117: oid 50 updating version 54 to 56 2023-12-15T22:56:54.492 INFO:tasks.rados.rados.0.smithi071.stdout:117: oid 50 version 56 is already newer than 55 2023-12-15T22:56:54.492 INFO:tasks.rados.rados.0.smithi071.stdout:117: oid 50 updating version 56 to 57 2023-12-15T22:56:54.492 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 57 (ObjNum 79 snap 6 seq_num 79) dirty exists 2023-12-15T22:56:54.492 INFO:tasks.rados.rados.0.smithi071.stdout:117: left oid 50 (ObjNum 79 snap 6 seq_num 79) 2023-12-15T22:56:54.492 INFO:tasks.rados.rados.0.smithi071.stdout:130: read oid 46 snap -1 2023-12-15T22:56:54.492 INFO:tasks.rados.rados.0.smithi071.stdout:130: expect (ObjNum 62 snap 2 seq_num 62) 2023-12-15T22:56:54.492 INFO:tasks.rados.rados.0.smithi071.stdout:117: done (11 left) 2023-12-15T22:56:54.492 INFO:tasks.rados.rados.0.smithi071.stdout:131: write oid 25 current snap is 6 2023-12-15T22:56:54.492 INFO:tasks.rados.rados.0.smithi071.stdout:119: finishing write tid 3 to smithi07183189-26 2023-12-15T22:56:54.492 INFO:tasks.rados.rados.0.smithi071.stdout:118: finishing write tid 3 to smithi07183189-49 2023-12-15T22:56:54.492 INFO:tasks.rados.rados.0.smithi071.stdout:119: finishing write tid 4 to smithi07183189-26 2023-12-15T22:56:54.492 INFO:tasks.rados.rados.0.smithi071.stdout:119: finishing write tid 5 to smithi07183189-26 2023-12-15T22:56:54.492 INFO:tasks.rados.rados.0.smithi071.stdout:119: oid 26 updating version 0 to 16 2023-12-15T22:56:54.492 INFO:tasks.rados.rados.0.smithi071.stdout:119: oid 26 version 16 is already newer than 15 2023-12-15T22:56:54.493 INFO:tasks.rados.rados.0.smithi071.stdout:119: oid 26 updating version 16 to 17 2023-12-15T22:56:54.493 INFO:tasks.rados.rados.0.smithi071.stdout:119: oid 26 version 17 is already newer than 14 2023-12-15T22:56:54.493 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 17 (ObjNum 81 snap 6 seq_num 81) dirty exists 2023-12-15T22:56:54.493 INFO:tasks.rados.rados.0.smithi071.stdout:119: left oid 26 (ObjNum 81 snap 6 seq_num 81) 2023-12-15T22:56:54.493 INFO:tasks.rados.rados.0.smithi071.stdout:120: finishing write tid 2 to smithi07183189-44 2023-12-15T22:56:54.493 INFO:tasks.rados.rados.0.smithi071.stdout:131: seq_num 86 ranges {645402=551297,1749569=492355,2845789=599613} 2023-12-15T22:56:54.497 INFO:tasks.rados.rados.0.smithi071.stdout:131: writing smithi07183189-25 from 645402 to 1196699 tid 1 2023-12-15T22:56:54.503 INFO:tasks.rados.rados.0.smithi071.stdout:131: writing smithi07183189-25 from 1749569 to 2241924 tid 2 2023-12-15T22:56:54.505 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2023-12-15T22:56:54.431101+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":108926937,"num_objects":62,"num_object_clones":15,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":372,"num_read_kb":125467,"num_write":419,"num_write_kb":79886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":381,"ondisk_log_size":381,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":279608,"kb_used_data":169724,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377207752,"statfs":{"total":386547056640,"available":386260738048,"internally_reserved":0,"allocated":173797376,"data_stored":173984353,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3109229,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":43,"num_read_kb":33968,"num_write":13,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.011058"},"pg_stats":[{"pgid":"3.f","version":"17'6","reported_seq":38,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-12-15T22:56:50.428734+0000","last_change":"2023-12-15T22:56:48.466656+0000","last_active":"2023-12-15T22:56:50.428734+0000","last_peered":"2023-12-15T22:56:50.428734+0000","last_clean":"2023-12-15T22:56:50.428734+0000","last_became_active":"2023-12-15T22:56:39.503797+0000","last_became_peered":"2023-12-15T22:56:39.503797+0000","last_unstale":"2023-12-15T22:56:50.428734+0000","last_undegraded":"2023-12-15T22:56:50.428734+0000","last_fullsized":"2023-12-15T22:56:50.428734+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4537824,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-15T22:56:49.432805+0000","last_change":"2023-12-15T22:56:48.461498+0000","last_active":"2023-12-15T22:56:49.432805+0000","last_peered":"2023-12-15T22:56:49.432805+0000","last_clean":"2023-12-15T22:56:49.432805+0000","last_became_active":"2023-12-15T22:56:39.552856+0000","last_became_peered":"2023-12-15T22:56:39.552856+0000","last_unstale":"2023-12-15T22:56:49.432805+0000","last_undegraded":"2023-12-15T22:56:49.432805+0000","last_fullsized":"2023-12-15T22:56:49.432805+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"23'29","reported_seq":77,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-15T22:56:49.428523+0000","last_change":"2023-12-15T22:56:48.575352+0000","last_active":"2023-12-15T22:56:49.428523+0000","last_peered":"2023-12-15T22:56:49.428523+0000","last_clean":"2023-12-15T22:56:49.428523+0000","last_became_active":"2023-12-15T22:56:39.495490+0000","last_became_peered":"2023-12-15T22:56:39.495490+0000","last_unstale":"2023-12-15T22:56:49.428523+0000","last_undegraded":"2023-12-15T22:56:49.428523+0000","last_fullsized":"2023-12-15T22:56:49.428523+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15294172,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":45,"num_read_kb":25190,"num_write":32,"num_write_kb":8456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"17'12","reported_seq":52,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-12-15T22:56:50.428974+0000","last_change":"2023-12-15T22:56:48.498773+0000","last_active":"2023-12-15T22:56:50.428974+0000","last_peered":"2023-12-15T22:56:50.428974+0000","last_clean":"2023-12-15T22:56:50.428974+0000","last_became_active":"2023-12-15T22:56:39.579234+0000","last_became_peered":"2023-12-15T22:56:39.579234+0000","last_unstale":"2023-12-15T22:56:50.428974+0000","last_undegraded":"2023-12-15T22:56:50.428974+0000","last_fullsized":"2023-12-15T22:56:50.428974+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7764735,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":10426,"num_write":16,"num_write_kb":4333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"17'10","reported_seq":46,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-15T22:56:49.621269+0000","last_change":"2023-12-15T22:56:48.435680+0000","last_active":"2023-12-15T22:56:49.621269+0000","last_peered":"2023-12-15T22:56:49.621269+0000","last_clean":"2023-12-15T22:56:49.621269+0000","last_became_active":"2023-12-15T22:56:39.572176+0000","last_became_peered":"2023-12-15T22:56:39.572176+0000","last_unstale":"2023-12-15T22:56:49.621269+0000","last_undegraded":"2023-12-15T22:56:49.621269+0000","last_fullsized":"2023-12-15T22:56:49.621269+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4638666,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":3684,"num_write":12,"num_write_kb":3492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"17'10","reported_seq":43,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-12-15T22:56:50.432486+0000","last_change":"2023-12-15T22:56:48.465864+0000","last_active":"2023-12-15T22:56:50.432486+0000","last_peered":"2023-12-15T22:56:50.432486+0000","last_clean":"2023-12-15T22:56:50.432486+0000","last_became_active":"2023-12-15T22:56:39.516361+0000","last_became_peered":"2023-12-15T22:56:39.516361+0000","last_unstale":"2023-12-15T22:56:50.432486+0000","last_undegraded":"2023-12-15T22:56:50.432486+0000","last_fullsized":"2023-12-15T22:56:50.432486+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5126069,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":13,"num_write_kb":3541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"23'53","reported_seq":88,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-12-15T22:56:50.434017+0000","last_change":"2023-12-15T22:56:48.606349+0000","last_active":"2023-12-15T22:56:50.434017+0000","last_peered":"2023-12-15T22:56:50.434017+0000","last_clean":"2023-12-15T22:56:50.434017+0000","last_became_active":"2023-12-15T22:56:39.519748+0000","last_became_peered":"2023-12-15T22:56:39.519748+0000","last_unstale":"2023-12-15T22:56:50.434017+0000","last_undegraded":"2023-12-15T22:56:50.434017+0000","last_fullsized":"2023-12-15T22:56:50.434017+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18756381,"num_objects":12,"num_object_clones":4,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":44,"num_read_kb":23720,"num_write":45,"num_write_kb":13495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"23'22","reported_seq":52,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-15T22:56:49.431395+0000","last_change":"2023-12-15T22:56:48.564071+0000","last_active":"2023-12-15T22:56:49.431395+0000","last_peered":"2023-12-15T22:56:49.431395+0000","last_clean":"2023-12-15T22:56:49.431395+0000","last_became_active":"2023-12-15T22:56:39.543179+0000","last_became_peered":"2023-12-15T22:56:39.543179+0000","last_unstale":"2023-12-15T22:56:49.431395+0000","last_undegraded":"2023-12-15T22:56:49.431395+0000","last_fullsized":"2023-12-15T22:56:49.431395+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8050454,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":22,"num_write_kb":5303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":160,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-15T22:56:49.623218+0000","last_change":"2023-12-15T22:56:32.607017+0000","last_active":"2023-12-15T22:56:49.623218+0000","last_peered":"2023-12-15T22:56:49.623218+0000","last_clean":"2023-12-15T22:56:49.623218+0000","last_became_active":"2023-12-15T22:56:32.602384+0000","last_became_peered":"2023-12-15T22:56:32.602384+0000","last_unstale":"2023-12-15T22:56:49.623218+0000","last_undegraded":"2023-12-15T22:56:49.623218+0000","last_fullsized":"2023-12-15T22:56:49.623218+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_clean_scrub_stamp":"2023-12-15T22:56:31.525572+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-15T22:56:49.428896+0000","last_change":"2023-12-15T22:56:33.627481+0000","last_active":"2023-12-15T22:56:49.428896+0000","last_peered":"2023-12-15T22:56:49.428896+0000","last_clean":"2023-12-15T22:56:49.428896+0000","last_became_active":"2023-12-15T22:56:33.624689+0000","last_became_peered":"2023-12-15T22:56:33.624689+0000","last_unstale":"2023-12-15T22:56:49.428896+0000","last_undegraded":"2023-12-15T22:56:49.428896+0000","last_fullsized":"2023-12-15T22:56:49.428896+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"23'16","reported_seq":58,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-15T22:56:49.428547+0000","last_change":"2023-12-15T22:56:48.458619+0000","last_active":"2023-12-15T22:56:49.428547+0000","last_peered":"2023-12-15T22:56:49.428547+0000","last_clean":"2023-12-15T22:56:49.428547+0000","last_became_active":"2023-12-15T22:56:39.529359+0000","last_became_peered":"2023-12-15T22:56:39.529359+0000","last_unstale":"2023-12-15T22:56:49.428547+0000","last_undegraded":"2023-12-15T22:56:49.428547+0000","last_fullsized":"2023-12-15T22:56:49.428547+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6554631,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":24,"num_read_kb":10696,"num_write":20,"num_write_kb":5567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":33,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-12-15T22:56:50.433297+0000","last_change":"2023-12-15T22:56:33.655797+0000","last_active":"2023-12-15T22:56:50.433297+0000","last_peered":"2023-12-15T22:56:50.433297+0000","last_clean":"2023-12-15T22:56:50.433297+0000","last_became_active":"2023-12-15T22:56:33.652885+0000","last_became_peered":"2023-12-15T22:56:33.652885+0000","last_unstale":"2023-12-15T22:56:50.433297+0000","last_undegraded":"2023-12-15T22:56:50.433297+0000","last_fullsized":"2023-12-15T22:56:50.433297+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"23'26","reported_seq":66,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-15T22:56:49.616731+0000","last_change":"2023-12-15T22:56:48.553682+0000","last_active":"2023-12-15T22:56:49.616731+0000","last_peered":"2023-12-15T22:56:49.616731+0000","last_clean":"2023-12-15T22:56:49.616731+0000","last_became_active":"2023-12-15T22:56:39.543509+0000","last_became_peered":"2023-12-15T22:56:39.543509+0000","last_unstale":"2023-12-15T22:56:49.616731+0000","last_undegraded":"2023-12-15T22:56:49.616731+0000","last_fullsized":"2023-12-15T22:56:49.616731+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7714314,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":31,"num_read_kb":15193,"num_write":25,"num_write_kb":6249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-15T22:56:49.430843+0000","last_change":"2023-12-15T22:56:33.600485+0000","last_active":"2023-12-15T22:56:49.430843+0000","last_peered":"2023-12-15T22:56:49.430843+0000","last_clean":"2023-12-15T22:56:49.430843+0000","last_became_active":"2023-12-15T22:56:33.597320+0000","last_became_peered":"2023-12-15T22:56:33.597320+0000","last_unstale":"2023-12-15T22:56:49.430843+0000","last_undegraded":"2023-12-15T22:56:49.430843+0000","last_fullsized":"2023-12-15T22:56:49.430843+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"23'21","reported_seq":53,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-15T22:56:49.614925+0000","last_change":"2023-12-15T22:56:48.562898+0000","last_active":"2023-12-15T22:56:49.614925+0000","last_peered":"2023-12-15T22:56:49.614925+0000","last_clean":"2023-12-15T22:56:49.614925+0000","last_became_active":"2023-12-15T22:56:39.560980+0000","last_became_peered":"2023-12-15T22:56:39.560980+0000","last_unstale":"2023-12-15T22:56:49.614925+0000","last_undegraded":"2023-12-15T22:56:49.614925+0000","last_fullsized":"2023-12-15T22:56:49.614925+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6403401,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":13739,"num_write":17,"num_write_kb":4752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":33,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-12-15T22:56:50.433195+0000","last_change":"2023-12-15T22:56:33.648541+0000","last_active":"2023-12-15T22:56:50.433195+0000","last_peered":"2023-12-15T22:56:50.433195+0000","last_clean":"2023-12-15T22:56:50.433195+0000","last_became_active":"2023-12-15T22:56:33.643681+0000","last_became_peered":"2023-12-15T22:56:33.643681+0000","last_unstale":"2023-12-15T22:56:50.433195+0000","last_undegraded":"2023-12-15T22:56:50.433195+0000","last_fullsized":"2023-12-15T22:56:50.433195+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"23'35","reported_seq":64,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-15T22:56:49.434238+0000","last_change":"2023-12-15T22:56:48.641308+0000","last_active":"2023-12-15T22:56:49.434238+0000","last_peered":"2023-12-15T22:56:49.434238+0000","last_clean":"2023-12-15T22:56:49.434238+0000","last_became_active":"2023-12-15T22:56:39.506510+0000","last_became_peered":"2023-12-15T22:56:39.506510+0000","last_unstale":"2023-12-15T22:56:49.434238+0000","last_undegraded":"2023-12-15T22:56:49.434238+0000","last_fullsized":"2023-12-15T22:56:49.434238+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9479049,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":18,"num_read_kb":4903,"num_write":30,"num_write_kb":8714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"17'7","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-15T22:56:49.429947+0000","last_change":"2023-12-15T22:56:48.475642+0000","last_active":"2023-12-15T22:56:49.429947+0000","last_peered":"2023-12-15T22:56:49.429947+0000","last_clean":"2023-12-15T22:56:49.429947+0000","last_became_active":"2023-12-15T22:56:39.511159+0000","last_became_peered":"2023-12-15T22:56:39.511159+0000","last_unstale":"2023-12-15T22:56:49.429947+0000","last_undegraded":"2023-12-15T22:56:49.429947+0000","last_fullsized":"2023-12-15T22:56:49.429947+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2941192,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":9,"num_write_kb":2133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-15T22:56:49.431804+0000","last_change":"2023-12-15T22:56:33.633890+0000","last_active":"2023-12-15T22:56:49.431804+0000","last_peered":"2023-12-15T22:56:49.431804+0000","last_clean":"2023-12-15T22:56:49.431804+0000","last_became_active":"2023-12-15T22:56:33.631134+0000","last_became_peered":"2023-12-15T22:56:33.631134+0000","last_unstale":"2023-12-15T22:56:49.431804+0000","last_undegraded":"2023-12-15T22:56:49.431804+0000","last_fullsized":"2023-12-15T22:56:49.431804+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"22'23","reported_seq":47,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-15T22:56:49.427989+0000","last_change":"2023-12-15T22:56:48.461137+0000","last_active":"2023-12-15T22:56:49.427989+0000","last_peered":"2023-12-15T22:56:49.427989+0000","last_clean":"2023-12-15T22:56:49.427989+0000","last_became_active":"2023-12-15T22:56:39.532730+0000","last_became_peered":"2023-12-15T22:56:39.532730+0000","last_unstale":"2023-12-15T22:56:49.427989+0000","last_undegraded":"2023-12-15T22:56:49.427989+0000","last_fullsized":"2023-12-15T22:56:49.427989+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6100875,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":17,"num_write_kb":5473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-15T22:56:49.620035+0000","last_change":"2023-12-15T22:56:33.655736+0000","last_active":"2023-12-15T22:56:49.620035+0000","last_peered":"2023-12-15T22:56:49.620035+0000","last_clean":"2023-12-15T22:56:49.620035+0000","last_became_active":"2023-12-15T22:56:33.652917+0000","last_became_peered":"2023-12-15T22:56:33.652917+0000","last_unstale":"2023-12-15T22:56:49.620035+0000","last_undegraded":"2023-12-15T22:56:49.620035+0000","last_fullsized":"2023-12-15T22:56:49.620035+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"17'11","reported_seq":50,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-15T22:56:49.621771+0000","last_change":"2023-12-15T22:56:48.438398+0000","last_active":"2023-12-15T22:56:49.621771+0000","last_peered":"2023-12-15T22:56:49.621771+0000","last_clean":"2023-12-15T22:56:49.621771+0000","last_became_active":"2023-12-15T22:56:39.557252+0000","last_became_peered":"2023-12-15T22:56:39.557252+0000","last_unstale":"2023-12-15T22:56:49.621771+0000","last_undegraded":"2023-12-15T22:56:49.621771+0000","last_fullsized":"2023-12-15T22:56:49.621771+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2420175,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":10323,"num_write":12,"num_write_kb":3292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-15T22:56:49.430425+0000","last_change":"2023-12-15T22:56:33.632348+0000","last_active":"2023-12-15T22:56:49.430425+0000","last_peered":"2023-12-15T22:56:49.430425+0000","last_clean":"2023-12-15T22:56:49.430425+0000","last_became_active":"2023-12-15T22:56:33.629022+0000","last_became_peered":"2023-12-15T22:56:33.629022+0000","last_unstale":"2023-12-15T22:56:49.430425+0000","last_undegraded":"2023-12-15T22:56:49.430425+0000","last_fullsized":"2023-12-15T22:56:49.430425+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"17'3","reported_seq":37,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-15T22:56:49.429912+0000","last_change":"2023-12-15T22:56:48.461758+0000","last_active":"2023-12-15T22:56:49.429912+0000","last_peered":"2023-12-15T22:56:49.429912+0000","last_clean":"2023-12-15T22:56:49.429912+0000","last_became_active":"2023-12-15T22:56:39.535145+0000","last_became_peered":"2023-12-15T22:56:39.535145+0000","last_unstale":"2023-12-15T22:56:49.429912+0000","last_undegraded":"2023-12-15T22:56:49.429912+0000","last_fullsized":"2023-12-15T22:56:49.429912+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2554631,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":7486,"num_write":4,"num_write_kb":1011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-15T22:56:49.430458+0000","last_change":"2023-12-15T22:56:33.627005+0000","last_active":"2023-12-15T22:56:49.430458+0000","last_peered":"2023-12-15T22:56:49.430458+0000","last_clean":"2023-12-15T22:56:49.430458+0000","last_became_active":"2023-12-15T22:56:33.622181+0000","last_became_peered":"2023-12-15T22:56:33.622181+0000","last_unstale":"2023-12-15T22:56:49.430458+0000","last_undegraded":"2023-12-15T22:56:49.430458+0000","last_fullsized":"2023-12-15T22:56:49.430458+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":108336569,"num_objects":60,"num_object_clones":15,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":269,"num_read_kb":125376,"num_write":282,"num_write_kb":77992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":284,"ondisk_log_size":284,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705685,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":72104,"kb_used_data":43748,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94299736,"statfs":{"total":96636764160,"available":96562929664,"internally_reserved":0,"allocated":44797952,"data_stored":44906506,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705683,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":65352,"kb_used_data":37472,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94306488,"statfs":{"total":96636764160,"available":96569843712,"internally_reserved":0,"allocated":38371328,"data_stored":38741239,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705683,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":69040,"kb_used_data":42216,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94302800,"statfs":{"total":96636764160,"available":96566067200,"internally_reserved":0,"allocated":43229184,"data_stored":42830261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705685,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":73112,"kb_used_data":46288,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94298728,"statfs":{"total":96636764160,"available":96561897472,"internally_reserved":0,"allocated":47398912,"data_stored":47506347,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-15T22:56:54.510 INFO:tasks.rados.rados.0.smithi071.stdout:131: writing smithi07183189-25 from 2845789 to 3445402 tid 3 2023-12-15T22:56:54.510 INFO:tasks.rados.rados.0.smithi071.stdout:119: done (11 left) 2023-12-15T22:56:54.510 INFO:tasks.rados.rados.0.smithi071.stdout:132: read oid 48 snap -1 2023-12-15T22:56:54.510 INFO:tasks.rados.rados.0.smithi071.stdout:132: expect deleted 2023-12-15T22:56:54.510 INFO:tasks.rados.rados.0.smithi071.stdout:133: write oid 11 current snap is 6 2023-12-15T22:56:54.510 INFO:tasks.rados.rados.0.smithi071.stdout:133: seq_num 87 ranges {662209=626546,1808051=636013,2873212=588997} 2023-12-15T22:56:54.517 INFO:tasks.rados.rados.0.smithi071.stdout:133: writing smithi07183189-11 from 662209 to 1288755 tid 1 2023-12-15T22:56:54.525 INFO:tasks.rados.rados.0.smithi071.stdout:133: writing smithi07183189-11 from 1808051 to 2444064 tid 2 2023-12-15T22:56:54.531 INFO:tasks.rados.rados.0.smithi071.stdout:133: writing smithi07183189-11 from 2873212 to 3462209 tid 3 2023-12-15T22:56:54.531 INFO:tasks.rados.rados.0.smithi071.stdout:134: snap_create 2023-12-15T22:56:54.531 INFO:tasks.rados.rados.0.smithi071.stdout:118: finishing write tid 4 to smithi07183189-49 2023-12-15T22:56:54.531 INFO:tasks.rados.rados.0.smithi071.stdout:116: finishing write tid 4 to smithi07183189-28 2023-12-15T22:56:54.531 INFO:tasks.rados.rados.0.smithi071.stdout:116: finishing write tid 5 to smithi07183189-28 2023-12-15T22:56:54.531 INFO:tasks.rados.rados.0.smithi071.stdout:116: oid 28 updating version 0 to 12 2023-12-15T22:56:54.532 INFO:tasks.rados.rados.0.smithi071.stdout:116: oid 28 updating version 12 to 15 2023-12-15T22:56:54.532 INFO:tasks.rados.rados.0.smithi071.stdout:116: oid 28 version 15 is already newer than 14 2023-12-15T22:56:54.532 INFO:tasks.rados.rados.0.smithi071.stdout:116: oid 28 version 15 is already newer than 13 2023-12-15T22:56:54.532 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 15 (ObjNum 78 snap 6 seq_num 78) dirty exists 2023-12-15T22:56:54.532 INFO:tasks.rados.rados.0.smithi071.stdout:116: left oid 28 (ObjNum 78 snap 6 seq_num 78) 2023-12-15T22:56:54.532 INFO:tasks.rados.rados.0.smithi071.stdout:118: finishing write tid 5 to smithi07183189-49 2023-12-15T22:56:54.532 INFO:tasks.rados.rados.0.smithi071.stdout:118: oid 49 updating version 0 to 37 2023-12-15T22:56:54.532 INFO:tasks.rados.rados.0.smithi071.stdout:118: oid 49 updating version 37 to 38 2023-12-15T22:56:54.532 INFO:tasks.rados.rados.0.smithi071.stdout:118: oid 49 updating version 38 to 39 2023-12-15T22:56:54.532 INFO:tasks.rados.rados.0.smithi071.stdout:118: oid 49 version 39 is already newer than 36 2023-12-15T22:56:54.532 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 49 v 39 (ObjNum 80 snap 6 seq_num 80) dirty exists 2023-12-15T22:56:54.532 INFO:tasks.rados.rados.0.smithi071.stdout:118: left oid 49 (ObjNum 80 snap 6 seq_num 80) 2023-12-15T22:56:54.532 INFO:tasks.rados.rados.0.smithi071.stdout:120: finishing write tid 3 to smithi07183189-44 2023-12-15T22:56:54.532 INFO:tasks.rados.rados.0.smithi071.stdout:121: expect (ObjNum 68 snap 3 seq_num 68) 2023-12-15T22:56:54.716 INFO:tasks.rados.rados.0.smithi071.stdout:120: finishing write tid 4 to smithi07183189-44 2023-12-15T22:56:54.716 INFO:tasks.rados.rados.0.smithi071.stdout:120: finishing write tid 5 to smithi07183189-44 2023-12-15T22:56:54.717 INFO:tasks.rados.rados.0.smithi071.stdout:120: oid 44 updating version 0 to 26 2023-12-15T22:56:54.717 INFO:tasks.rados.rados.0.smithi071.stdout:120: oid 44 updating version 26 to 28 2023-12-15T22:56:54.717 INFO:tasks.rados.rados.0.smithi071.stdout:120: oid 44 version 28 is already newer than 25 2023-12-15T22:56:54.717 INFO:tasks.rados.rados.0.smithi071.stdout:120: oid 44 version 28 is already newer than 27 2023-12-15T22:56:54.717 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 28 (ObjNum 82 snap 6 seq_num 82) dirty exists 2023-12-15T22:56:54.717 INFO:tasks.rados.rados.0.smithi071.stdout:120: left oid 44 (ObjNum 82 snap 6 seq_num 82) 2023-12-15T22:56:54.717 INFO:tasks.rados.rados.0.smithi071.stdout:123: expect (ObjNum 36 snap 0 seq_num 36) 2023-12-15T22:56:54.821 INFO:tasks.rados.rados.0.smithi071.stdout:130: expect (ObjNum 62 snap 2 seq_num 62) 2023-12-15T22:56:54.979 INFO:tasks.rados.rados.0.smithi071.stdout:127: finishing write tid 1 to smithi07183189-35 2023-12-15T22:56:54.980 INFO:tasks.rados.rados.0.smithi071.stdout:126: finishing write tid 1 to smithi07183189-36 2023-12-15T22:56:54.980 INFO:tasks.rados.rados.0.smithi071.stdout:127: finishing write tid 2 to smithi07183189-35 2023-12-15T22:56:54.980 INFO:tasks.rados.rados.0.smithi071.stdout:133: finishing write tid 1 to smithi07183189-11 2023-12-15T22:56:54.980 INFO:tasks.rados.rados.0.smithi071.stdout:133: finishing write tid 2 to smithi07183189-11 2023-12-15T22:56:54.980 INFO:tasks.rados.rados.0.smithi071.stdout:128: finishing write tid 1 to smithi07183189-2 2023-12-15T22:56:54.980 INFO:tasks.rados.rados.0.smithi071.stdout:127: finishing write tid 3 to smithi07183189-35 2023-12-15T22:56:54.980 INFO:tasks.rados.rados.0.smithi071.stdout:126: finishing write tid 2 to smithi07183189-36 2023-12-15T22:56:54.980 INFO:tasks.rados.rados.0.smithi071.stdout:133: finishing write tid 3 to smithi07183189-11 2023-12-15T22:56:54.980 INFO:tasks.rados.rados.0.smithi071.stdout:128: finishing write tid 2 to smithi07183189-2 2023-12-15T22:56:54.980 INFO:tasks.rados.rados.0.smithi071.stdout:133: finishing write tid 4 to smithi07183189-11 2023-12-15T22:56:54.980 INFO:tasks.rados.rados.0.smithi071.stdout:133: finishing write tid 5 to smithi07183189-11 2023-12-15T22:56:54.980 INFO:tasks.rados.rados.0.smithi071.stdout:133: oid 11 updating version 0 to 23 2023-12-15T22:56:54.981 INFO:tasks.rados.rados.0.smithi071.stdout:133: oid 11 version 23 is already newer than 22 2023-12-15T22:56:54.981 INFO:tasks.rados.rados.0.smithi071.stdout:133: oid 11 version 23 is already newer than 22 2023-12-15T22:56:54.981 INFO:tasks.rados.rados.0.smithi071.stdout:133: oid 11 updating version 23 to 24 2023-12-15T22:56:54.981 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 11 v 24 (ObjNum 87 snap 6 seq_num 87) dirty exists 2023-12-15T22:56:54.981 INFO:tasks.rados.rados.0.smithi071.stdout:133: left oid 11 (ObjNum 87 snap 6 seq_num 87) 2023-12-15T22:56:54.981 INFO:tasks.rados.rados.0.smithi071.stdout:127: finishing write tid 4 to smithi07183189-35 2023-12-15T22:56:54.981 INFO:tasks.rados.rados.0.smithi071.stdout:127: finishing write tid 5 to smithi07183189-35 2023-12-15T22:56:54.981 INFO:tasks.rados.rados.0.smithi071.stdout:127: oid 35 updating version 0 to 13 2023-12-15T22:56:54.981 INFO:tasks.rados.rados.0.smithi071.stdout:127: oid 35 updating version 13 to 14 2023-12-15T22:56:54.981 INFO:tasks.rados.rados.0.smithi071.stdout:127: oid 35 updating version 14 to 16 2023-12-15T22:56:54.981 INFO:tasks.rados.rados.0.smithi071.stdout:127: oid 35 version 16 is already newer than 15 2023-12-15T22:56:54.981 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 16 (ObjNum 84 snap 6 seq_num 84) dirty exists 2023-12-15T22:56:54.981 INFO:tasks.rados.rados.0.smithi071.stdout:127: left oid 35 (ObjNum 84 snap 6 seq_num 84) 2023-12-15T22:56:54.981 INFO:tasks.rados.rados.0.smithi071.stdout:128: finishing write tid 3 to smithi07183189-2 2023-12-15T22:56:54.982 INFO:tasks.rados.rados.0.smithi071.stdout:126: finishing write tid 3 to smithi07183189-36 2023-12-15T22:56:54.982 INFO:tasks.rados.rados.0.smithi071.stdout:126: finishing write tid 4 to smithi07183189-36 2023-12-15T22:56:54.982 INFO:tasks.rados.rados.0.smithi071.stdout:126: finishing write tid 5 to smithi07183189-36 2023-12-15T22:56:54.982 INFO:tasks.rados.rados.0.smithi071.stdout:126: oid 36 updating version 0 to 30 2023-12-15T22:56:54.982 INFO:tasks.rados.rados.0.smithi071.stdout:126: oid 36 updating version 30 to 33 2023-12-15T22:56:54.982 INFO:tasks.rados.rados.0.smithi071.stdout:126: oid 36 version 33 is already newer than 31 2023-12-15T22:56:54.982 INFO:tasks.rados.rados.0.smithi071.stdout:126: oid 36 version 33 is already newer than 33 2023-12-15T22:56:54.982 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 33 (ObjNum 83 snap 6 seq_num 83) dirty exists 2023-12-15T22:56:54.982 INFO:tasks.rados.rados.0.smithi071.stdout:126: left oid 36 (ObjNum 83 snap 6 seq_num 83) 2023-12-15T22:56:54.982 INFO:tasks.rados.rados.0.smithi071.stdout:129: expect (ObjNum 53 snap 0 seq_num 53) 2023-12-15T22:56:55.130 INFO:tasks.rados.rados.0.smithi071.stdout:131: finishing write tid 1 to smithi07183189-25 2023-12-15T22:56:55.130 INFO:tasks.rados.rados.0.smithi071.stdout:131: finishing write tid 2 to smithi07183189-25 2023-12-15T22:56:55.130 INFO:tasks.rados.rados.0.smithi071.stdout:128: finishing write tid 4 to smithi07183189-2 2023-12-15T22:56:55.130 INFO:tasks.rados.rados.0.smithi071.stdout:128: finishing write tid 5 to smithi07183189-2 2023-12-15T22:56:55.130 INFO:tasks.rados.rados.0.smithi071.stdout:128: oid 2 updating version 0 to 14 2023-12-15T22:56:55.130 INFO:tasks.rados.rados.0.smithi071.stdout:128: oid 2 version 14 is already newer than 13 2023-12-15T22:56:55.130 INFO:tasks.rados.rados.0.smithi071.stdout:128: oid 2 version 14 is already newer than 12 2023-12-15T22:56:55.130 INFO:tasks.rados.rados.0.smithi071.stdout:128: oid 2 version 14 is already newer than 11 2023-12-15T22:56:55.130 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 14 (ObjNum 85 snap 6 seq_num 85) dirty exists 2023-12-15T22:56:55.130 INFO:tasks.rados.rados.0.smithi071.stdout:128: left oid 2 (ObjNum 85 snap 6 seq_num 85) 2023-12-15T22:56:55.130 INFO:tasks.rados.rados.0.smithi071.stdout:131: finishing write tid 3 to smithi07183189-25 2023-12-15T22:56:55.131 INFO:tasks.rados.rados.0.smithi071.stdout:131: finishing write tid 4 to smithi07183189-25 2023-12-15T22:56:55.131 INFO:tasks.rados.rados.0.smithi071.stdout:131: finishing write tid 5 to smithi07183189-25 2023-12-15T22:56:55.131 INFO:tasks.rados.rados.0.smithi071.stdout:131: oid 25 updating version 0 to 37 2023-12-15T22:56:55.131 INFO:tasks.rados.rados.0.smithi071.stdout:131: oid 25 version 37 is already newer than 35 2023-12-15T22:56:55.131 INFO:tasks.rados.rados.0.smithi071.stdout:131: oid 25 version 37 is already newer than 36 2023-12-15T22:56:55.131 INFO:tasks.rados.rados.0.smithi071.stdout:131: oid 25 version 37 is already newer than 35 2023-12-15T22:56:55.131 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 25 v 37 (ObjNum 86 snap 6 seq_num 86) dirty exists 2023-12-15T22:56:55.131 INFO:tasks.rados.rados.0.smithi071.stdout:131: left oid 25 (ObjNum 86 snap 6 seq_num 86) 2023-12-15T22:56:55.382 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:56:55.440 INFO:tasks.rados.rados.0.smithi071.stdout:116: done (13 left) 2023-12-15T22:56:55.440 INFO:tasks.rados.rados.0.smithi071.stdout:118: done (12 left) 2023-12-15T22:56:55.440 INFO:tasks.rados.rados.0.smithi071.stdout:120: done (11 left) 2023-12-15T22:56:55.440 INFO:tasks.rados.rados.0.smithi071.stdout:121: done (10 left) 2023-12-15T22:56:55.440 INFO:tasks.rados.rados.0.smithi071.stdout:123: done (9 left) 2023-12-15T22:56:55.440 INFO:tasks.rados.rados.0.smithi071.stdout:126: done (8 left) 2023-12-15T22:56:55.440 INFO:tasks.rados.rados.0.smithi071.stdout:127: done (7 left) 2023-12-15T22:56:55.440 INFO:tasks.rados.rados.0.smithi071.stdout:128: done (6 left) 2023-12-15T22:56:55.440 INFO:tasks.rados.rados.0.smithi071.stdout:129: done (5 left) 2023-12-15T22:56:55.441 INFO:tasks.rados.rados.0.smithi071.stdout:130: done (4 left) 2023-12-15T22:56:55.441 INFO:tasks.rados.rados.0.smithi071.stdout:131: done (3 left) 2023-12-15T22:56:55.441 INFO:tasks.rados.rados.0.smithi071.stdout:132: done (2 left) 2023-12-15T22:56:55.441 INFO:tasks.rados.rados.0.smithi071.stdout:133: done (1 left) 2023-12-15T22:56:55.441 INFO:tasks.rados.rados.0.smithi071.stdout:134: done (0 left) 2023-12-15T22:56:55.441 INFO:tasks.rados.rados.0.smithi071.stdout:135: write (excl) oid 42 current snap is 7 2023-12-15T22:56:55.441 INFO:tasks.rados.rados.0.smithi071.stdout:135: seq_num 88 ranges {679016=701795,1866533=496024,2900635=402643} 2023-12-15T22:56:55.448 INFO:tasks.rados.rados.0.smithi071.stdout:135: writing smithi07183189-42 from 679016 to 1380811 tid 1 2023-12-15T22:56:55.454 INFO:tasks.rados.rados.0.smithi071.stdout:135: writing smithi07183189-42 from 1866533 to 2362557 tid 2 2023-12-15T22:56:55.460 INFO:tasks.rados.rados.0.smithi071.stdout:135: writing smithi07183189-42 from 2900635 to 3303278 tid 3 2023-12-15T22:56:55.460 INFO:tasks.rados.rados.0.smithi071.stdout:136: snap_remove snap 6 2023-12-15T22:56:55.490 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:55.489+0000 7ff8bf62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:55.494 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:55.494+0000 7ff8bf62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:56.448 INFO:tasks.rados.rados.0.smithi071.stdout:136: done (1 left) 2023-12-15T22:56:56.448 INFO:tasks.rados.rados.0.smithi071.stdout:137: read oid 2 snap -1 2023-12-15T22:56:56.448 INFO:tasks.rados.rados.0.smithi071.stdout:137: expect (ObjNum 85 snap 6 seq_num 85) 2023-12-15T22:56:56.448 INFO:tasks.rados.rados.0.smithi071.stdout:138: read oid 16 snap -1 2023-12-15T22:56:56.448 INFO:tasks.rados.rados.0.smithi071.stdout:138: expect (ObjNum 51 snap 0 seq_num 51) 2023-12-15T22:56:56.448 INFO:tasks.rados.rados.0.smithi071.stdout:139: read oid 32 snap -1 2023-12-15T22:56:56.448 INFO:tasks.rados.rados.0.smithi071.stdout:139: expect (ObjNum 31 snap 0 seq_num 31) 2023-12-15T22:56:56.449 INFO:tasks.rados.rados.0.smithi071.stdout:140: read oid 22 snap -1 2023-12-15T22:56:56.449 INFO:tasks.rados.rados.0.smithi071.stdout:140: expect (ObjNum 68 snap 3 seq_num 68) 2023-12-15T22:56:56.449 INFO:tasks.rados.rados.0.smithi071.stdout:141: write oid 17 current snap is 7 2023-12-15T22:56:56.449 INFO:tasks.rados.rados.0.smithi071.stdout:135: finishing write tid 1 to smithi07183189-42 2023-12-15T22:56:56.449 INFO:tasks.rados.rados.0.smithi071.stdout:141: seq_num 89 ranges {695823=777044,1925015=639682,3328058=167765} 2023-12-15T22:56:56.456 INFO:tasks.rados.rados.0.smithi071.stdout:141: writing smithi07183189-17 from 695823 to 1472867 tid 1 2023-12-15T22:56:56.462 INFO:tasks.rados.rados.0.smithi071.stdout:141: writing smithi07183189-17 from 1925015 to 2564697 tid 2 2023-12-15T22:56:56.468 INFO:tasks.rados.rados.0.smithi071.stdout:141: writing smithi07183189-17 from 3328058 to 3495823 tid 3 2023-12-15T22:56:56.468 INFO:tasks.rados.rados.0.smithi071.stdout:142: write (excl) oid 36 current snap is 7 2023-12-15T22:56:56.469 INFO:tasks.rados.rados.0.smithi071.stdout:135: finishing write tid 2 to smithi07183189-42 2023-12-15T22:56:56.469 INFO:tasks.rados.rados.0.smithi071.stdout:142: seq_num 90 ranges {712630=452293,1583497=499693,2555481=425187} 2023-12-15T22:56:56.476 INFO:tasks.rados.rados.0.smithi071.stdout:142: writing smithi07183189-36 from 712630 to 1164923 tid 1 2023-12-15T22:56:56.482 INFO:tasks.rados.rados.0.smithi071.stdout:142: writing smithi07183189-36 from 1583497 to 2083190 tid 2 2023-12-15T22:56:56.489 INFO:tasks.rados.rados.0.smithi071.stdout:142: writing smithi07183189-36 from 2555481 to 2980668 tid 3 2023-12-15T22:56:56.489 INFO:tasks.rados.rados.0.smithi071.stdout:143: write (excl) oid 9 current snap is 7 2023-12-15T22:56:56.489 INFO:tasks.rados.rados.0.smithi071.stdout:135: finishing write tid 3 to smithi07183189-42 2023-12-15T22:56:56.489 INFO:tasks.rados.rados.0.smithi071.stdout:135: finishing write tid 4 to smithi07183189-42 2023-12-15T22:56:56.490 INFO:tasks.rados.rados.0.smithi071.stdout:143: seq_num 91 ranges {729437=527542,1925626=643351,3266551=262886} 2023-12-15T22:56:56.497 INFO:tasks.rados.rados.0.smithi071.stdout:143: writing smithi07183189-9 from 729437 to 1256979 tid 1 2023-12-15T22:56:56.506 INFO:tasks.rados.rados.0.smithi071.stdout:143: writing smithi07183189-9 from 1925626 to 2568977 tid 2 2023-12-15T22:56:56.512 INFO:tasks.rados.rados.0.smithi071.stdout:143: writing smithi07183189-9 from 3266551 to 3529437 tid 3 2023-12-15T22:56:56.512 INFO:tasks.rados.rados.0.smithi071.stdout:144: write oid 11 current snap is 7 2023-12-15T22:56:56.512 INFO:tasks.rados.rados.0.smithi071.stdout:135: finishing write tid 5 to smithi07183189-42 2023-12-15T22:56:56.512 INFO:tasks.rados.rados.0.smithi071.stdout:135: oid 42 updating version 0 to 28 2023-12-15T22:56:56.512 INFO:tasks.rados.rados.0.smithi071.stdout:135: oid 42 version 28 is already newer than 27 2023-12-15T22:56:56.512 INFO:tasks.rados.rados.0.smithi071.stdout:135: oid 42 version 28 is already newer than 25 2023-12-15T22:56:56.512 INFO:tasks.rados.rados.0.smithi071.stdout:135: oid 42 version 28 is already newer than 27 2023-12-15T22:56:56.512 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 28 (ObjNum 88 snap 7 seq_num 88) dirty exists 2023-12-15T22:56:56.513 INFO:tasks.rados.rados.0.smithi071.stdout:135: left oid 42 (ObjNum 88 snap 7 seq_num 88) 2023-12-15T22:56:56.513 INFO:tasks.rados.rados.0.smithi071.stdout:144: seq_num 92 ranges {746244=719144,2100461=503362,3126680=419564} 2023-12-15T22:56:56.520 INFO:tasks.rados.rados.0.smithi071.stdout:144: writing smithi07183189-11 from 746244 to 1465388 tid 1 2023-12-15T22:56:56.527 INFO:tasks.rados.rados.0.smithi071.stdout:144: writing smithi07183189-11 from 2100461 to 2603823 tid 2 2023-12-15T22:56:56.532 INFO:tasks.rados.rados.0.smithi071.stdout:144: writing smithi07183189-11 from 3126680 to 3546244 tid 3 2023-12-15T22:56:56.532 INFO:tasks.rados.rados.0.smithi071.stdout:135: done (8 left) 2023-12-15T22:56:56.532 INFO:tasks.rados.rados.0.smithi071.stdout:145: write oid 14 current snap is 7 2023-12-15T22:56:56.533 INFO:tasks.rados.rados.0.smithi071.stdout:145: seq_num 93 ranges {763051=794393,2158943=647020,3437750=125301} 2023-12-15T22:56:56.542 INFO:tasks.rados.rados.0.smithi071.stdout:145: writing smithi07183189-14 from 763051 to 1557444 tid 1 2023-12-15T22:56:56.550 INFO:tasks.rados.rados.0.smithi071.stdout:145: writing smithi07183189-14 from 2158943 to 2805963 tid 2 2023-12-15T22:56:56.554 INFO:tasks.rados.rados.0.smithi071.stdout:145: writing smithi07183189-14 from 3437750 to 3563051 tid 3 2023-12-15T22:56:56.554 INFO:tasks.rados.rados.0.smithi071.stdout:146: write oid 31 current snap is 7 2023-12-15T22:56:56.554 INFO:tasks.rados.rados.0.smithi071.stdout:140: expect (ObjNum 68 snap 3 seq_num 68) 2023-12-15T22:56:56.784 INFO:tasks.rados.rados.0.smithi071.stdout:146: seq_num 94 ranges {779858=469642,1817425=507031,2781526=586628} 2023-12-15T22:56:56.791 INFO:tasks.rados.rados.0.smithi071.stdout:146: writing smithi07183189-31 from 779858 to 1249500 tid 1 2023-12-15T22:56:56.797 INFO:tasks.rados.rados.0.smithi071.stdout:146: writing smithi07183189-31 from 1817425 to 2324456 tid 2 2023-12-15T22:56:56.804 INFO:tasks.rados.rados.0.smithi071.stdout:146: writing smithi07183189-31 from 2781526 to 3368154 tid 3 2023-12-15T22:56:56.804 INFO:tasks.rados.rados.0.smithi071.stdout:140: done (9 left) 2023-12-15T22:56:56.804 INFO:tasks.rados.rados.0.smithi071.stdout:147: read oid 39 snap -1 2023-12-15T22:56:56.804 INFO:tasks.rados.rados.0.smithi071.stdout:147: expect (ObjNum 65 snap 2 seq_num 65) 2023-12-15T22:56:56.804 INFO:tasks.rados.rados.0.smithi071.stdout:148: read oid 33 snap -1 2023-12-15T22:56:56.804 INFO:tasks.rados.rados.0.smithi071.stdout:148: expect (ObjNum 32 snap 0 seq_num 32) 2023-12-15T22:56:56.804 INFO:tasks.rados.rados.0.smithi071.stdout:149: delete oid 50 current snap is 7 2023-12-15T22:56:56.804 INFO:tasks.rados.rados.0.smithi071.stdout:139: expect (ObjNum 31 snap 0 seq_num 31) 2023-12-15T22:56:56.918 INFO:tasks.rados.rados.0.smithi071.stdout:137: expect (ObjNum 85 snap 6 seq_num 85) 2023-12-15T22:56:57.101 INFO:tasks.rados.rados.0.smithi071.stdout:143: finishing write tid 1 to smithi07183189-9 2023-12-15T22:56:57.101 INFO:tasks.rados.rados.0.smithi071.stdout:138: expect (ObjNum 51 snap 0 seq_num 51) 2023-12-15T22:56:57.251 INFO:tasks.rados.rados.0.smithi071.stdout:145: finishing write tid 1 to smithi07183189-14 2023-12-15T22:56:57.251 INFO:tasks.rados.rados.0.smithi071.stdout:144: finishing write tid 1 to smithi07183189-11 2023-12-15T22:56:57.251 INFO:tasks.rados.rados.0.smithi071.stdout:143: finishing write tid 2 to smithi07183189-9 2023-12-15T22:56:57.251 INFO:tasks.rados.rados.0.smithi071.stdout:141: finishing write tid 1 to smithi07183189-17 2023-12-15T22:56:57.251 INFO:tasks.rados.rados.0.smithi071.stdout:144: finishing write tid 2 to smithi07183189-11 2023-12-15T22:56:57.252 INFO:tasks.rados.rados.0.smithi071.stdout:145: finishing write tid 2 to smithi07183189-14 2023-12-15T22:56:57.252 INFO:tasks.rados.rados.0.smithi071.stdout:143: finishing write tid 3 to smithi07183189-9 2023-12-15T22:56:57.252 INFO:tasks.rados.rados.0.smithi071.stdout:141: finishing write tid 2 to smithi07183189-17 2023-12-15T22:56:57.252 INFO:tasks.rados.rados.0.smithi071.stdout:145: finishing write tid 3 to smithi07183189-14 2023-12-15T22:56:57.252 INFO:tasks.rados.rados.0.smithi071.stdout:143: finishing write tid 4 to smithi07183189-9 2023-12-15T22:56:57.252 INFO:tasks.rados.rados.0.smithi071.stdout:144: finishing write tid 3 to smithi07183189-11 2023-12-15T22:56:57.252 INFO:tasks.rados.rados.0.smithi071.stdout:143: finishing write tid 5 to smithi07183189-9 2023-12-15T22:56:57.252 INFO:tasks.rados.rados.0.smithi071.stdout:143: oid 9 updating version 0 to 19 2023-12-15T22:56:57.252 INFO:tasks.rados.rados.0.smithi071.stdout:143: oid 9 updating version 19 to 20 2023-12-15T22:56:57.252 INFO:tasks.rados.rados.0.smithi071.stdout:143: oid 9 version 20 is already newer than 18 2023-12-15T22:56:57.252 INFO:tasks.rados.rados.0.smithi071.stdout:143: oid 9 version 20 is already newer than 17 2023-12-15T22:56:57.252 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 20 (ObjNum 91 snap 7 seq_num 91) dirty exists 2023-12-15T22:56:57.252 INFO:tasks.rados.rados.0.smithi071.stdout:143: left oid 9 (ObjNum 91 snap 7 seq_num 91) 2023-12-15T22:56:57.252 INFO:tasks.rados.rados.0.smithi071.stdout:141: finishing write tid 3 to smithi07183189-17 2023-12-15T22:56:57.253 INFO:tasks.rados.rados.0.smithi071.stdout:145: finishing write tid 4 to smithi07183189-14 2023-12-15T22:56:57.253 INFO:tasks.rados.rados.0.smithi071.stdout:137: done (11 left) 2023-12-15T22:56:57.253 INFO:tasks.rados.rados.0.smithi071.stdout:138: done (10 left) 2023-12-15T22:56:57.253 INFO:tasks.rados.rados.0.smithi071.stdout:139: done (9 left) 2023-12-15T22:56:57.253 INFO:tasks.rados.rados.0.smithi071.stdout:143: done (8 left) 2023-12-15T22:56:57.253 INFO:tasks.rados.rados.0.smithi071.stdout:149: done (7 left) 2023-12-15T22:56:57.253 INFO:tasks.rados.rados.0.smithi071.stdout:150: write (excl) oid 3 current snap is 7 2023-12-15T22:56:57.253 INFO:tasks.rados.rados.0.smithi071.stdout:145: finishing write tid 5 to smithi07183189-14 2023-12-15T22:56:57.253 INFO:tasks.rados.rados.0.smithi071.stdout:145: oid 14 updating version 0 to 29 2023-12-15T22:56:57.253 INFO:tasks.rados.rados.0.smithi071.stdout:145: oid 14 version 29 is already newer than 27 2023-12-15T22:56:57.253 INFO:tasks.rados.rados.0.smithi071.stdout:145: oid 14 version 29 is already newer than 28 2023-12-15T22:56:57.253 INFO:tasks.rados.rados.0.smithi071.stdout:145: oid 14 updating version 29 to 30 2023-12-15T22:56:57.253 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 30 (ObjNum 93 snap 7 seq_num 93) dirty exists 2023-12-15T22:56:57.253 INFO:tasks.rados.rados.0.smithi071.stdout:145: left oid 14 (ObjNum 93 snap 7 seq_num 93) 2023-12-15T22:56:57.253 INFO:tasks.rados.rados.0.smithi071.stdout:144: finishing write tid 4 to smithi07183189-11 2023-12-15T22:56:57.253 INFO:tasks.rados.rados.0.smithi071.stdout:144: finishing write tid 5 to smithi07183189-11 2023-12-15T22:56:57.254 INFO:tasks.rados.rados.0.smithi071.stdout:144: oid 11 updating version 0 to 29 2023-12-15T22:56:57.254 INFO:tasks.rados.rados.0.smithi071.stdout:144: oid 11 updating version 29 to 32 2023-12-15T22:56:57.254 INFO:tasks.rados.rados.0.smithi071.stdout:144: oid 11 version 32 is already newer than 31 2023-12-15T22:56:57.254 INFO:tasks.rados.rados.0.smithi071.stdout:144: oid 11 version 32 is already newer than 30 2023-12-15T22:56:57.254 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 11 v 32 (ObjNum 92 snap 7 seq_num 92) dirty exists 2023-12-15T22:56:57.254 INFO:tasks.rados.rados.0.smithi071.stdout:144: left oid 11 (ObjNum 92 snap 7 seq_num 92) 2023-12-15T22:56:57.254 INFO:tasks.rados.rados.0.smithi071.stdout:141: finishing write tid 4 to smithi07183189-17 2023-12-15T22:56:57.254 INFO:tasks.rados.rados.0.smithi071.stdout:150: seq_num 96 ranges {413472=620140,1818036=510700,3120019=493453} 2023-12-15T22:56:57.256 INFO:tasks.rados.rados.0.smithi071.stdout:150: writing smithi07183189-3 from 413472 to 1033612 tid 1 2023-12-15T22:56:57.264 INFO:tasks.rados.rados.0.smithi071.stdout:150: writing smithi07183189-3 from 1818036 to 2328736 tid 2 2023-12-15T22:56:57.271 INFO:tasks.rados.rados.0.smithi071.stdout:150: writing smithi07183189-3 from 3120019 to 3613472 tid 3 2023-12-15T22:56:57.272 INFO:tasks.rados.rados.0.smithi071.stdout:144: done (7 left) 2023-12-15T22:56:57.272 INFO:tasks.rados.rados.0.smithi071.stdout:145: done (6 left) 2023-12-15T22:56:57.272 INFO:tasks.rados.rados.0.smithi071.stdout:151: delete oid 40 current snap is 7 2023-12-15T22:56:57.272 INFO:tasks.rados.rados.0.smithi071.stdout:141: finishing write tid 5 to smithi07183189-17 2023-12-15T22:56:57.272 INFO:tasks.rados.rados.0.smithi071.stdout:141: oid 17 updating version 0 to 40 2023-12-15T22:56:57.272 INFO:tasks.rados.rados.0.smithi071.stdout:141: oid 17 version 40 is already newer than 38 2023-12-15T22:56:57.272 INFO:tasks.rados.rados.0.smithi071.stdout:141: oid 17 version 40 is already newer than 39 2023-12-15T22:56:57.272 INFO:tasks.rados.rados.0.smithi071.stdout:141: oid 17 updating version 40 to 41 2023-12-15T22:56:57.272 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 17 v 41 (ObjNum 89 snap 7 seq_num 89) dirty exists 2023-12-15T22:56:57.272 INFO:tasks.rados.rados.0.smithi071.stdout:141: left oid 17 (ObjNum 89 snap 7 seq_num 89) 2023-12-15T22:56:57.272 INFO:tasks.rados.rados.0.smithi071.stdout:142: finishing write tid 1 to smithi07183189-36 2023-12-15T22:56:57.272 INFO:tasks.rados.rados.0.smithi071.stdout:142: finishing write tid 2 to smithi07183189-36 2023-12-15T22:56:57.272 INFO:tasks.rados.rados.0.smithi071.stdout:147: expect (ObjNum 65 snap 2 seq_num 65) 2023-12-15T22:56:57.440 INFO:tasks.rados.rados.0.smithi071.stdout:146: finishing write tid 1 to smithi07183189-31 2023-12-15T22:56:57.440 INFO:tasks.rados.rados.0.smithi071.stdout:142: finishing write tid 3 to smithi07183189-36 2023-12-15T22:56:57.440 INFO:tasks.rados.rados.0.smithi071.stdout:141: done (6 left) 2023-12-15T22:56:57.440 INFO:tasks.rados.rados.0.smithi071.stdout:147: done (5 left) 2023-12-15T22:56:57.441 INFO:tasks.rados.rados.0.smithi071.stdout:151: done (4 left) 2023-12-15T22:56:57.441 INFO:tasks.rados.rados.0.smithi071.stdout:152: write (excl) oid 5 current snap is 7 2023-12-15T22:56:57.441 INFO:tasks.rados.rados.0.smithi071.stdout:142: finishing write tid 4 to smithi07183189-36 2023-12-15T22:56:57.441 INFO:tasks.rados.rados.0.smithi071.stdout:152: seq_num 98 ranges {447086=770638,1935000=798016,3458512=188574} 2023-12-15T22:56:57.447 INFO:tasks.rados.rados.0.smithi071.stdout:152: writing smithi07183189-5 from 447086 to 1217724 tid 1 2023-12-15T22:56:57.455 INFO:tasks.rados.rados.0.smithi071.stdout:152: writing smithi07183189-5 from 1935000 to 2733016 tid 2 2023-12-15T22:56:57.461 INFO:tasks.rados.rados.0.smithi071.stdout:152: writing smithi07183189-5 from 3458512 to 3647086 tid 3 2023-12-15T22:56:57.461 INFO:tasks.rados.rados.0.smithi071.stdout:153: read oid 50 snap -1 2023-12-15T22:56:57.461 INFO:tasks.rados.rados.0.smithi071.stdout:153: expect deleted 2023-12-15T22:56:57.461 INFO:tasks.rados.rados.0.smithi071.stdout:154: snap_create 2023-12-15T22:56:57.461 INFO:tasks.rados.rados.0.smithi071.stdout:142: finishing write tid 5 to smithi07183189-36 2023-12-15T22:56:57.461 INFO:tasks.rados.rados.0.smithi071.stdout:142: oid 36 updating version 0 to 42 2023-12-15T22:56:57.461 INFO:tasks.rados.rados.0.smithi071.stdout:142: oid 36 updating version 42 to 45 2023-12-15T22:56:57.461 INFO:tasks.rados.rados.0.smithi071.stdout:142: oid 36 version 45 is already newer than 45 2023-12-15T22:56:57.461 INFO:tasks.rados.rados.0.smithi071.stdout:142: oid 36 version 45 is already newer than 45 2023-12-15T22:56:57.461 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 45 (ObjNum 90 snap 7 seq_num 90) dirty exists 2023-12-15T22:56:57.462 INFO:tasks.rados.rados.0.smithi071.stdout:142: left oid 36 (ObjNum 90 snap 7 seq_num 90) 2023-12-15T22:56:57.462 INFO:tasks.rados.rados.0.smithi071.stdout:148: expect (ObjNum 32 snap 0 seq_num 32) 2023-12-15T22:56:57.573 INFO:tasks.rados.rados.0.smithi071.stdout:146: finishing write tid 2 to smithi07183189-31 2023-12-15T22:56:57.574 INFO:tasks.rados.rados.0.smithi071.stdout:146: finishing write tid 3 to smithi07183189-31 2023-12-15T22:56:57.574 INFO:tasks.rados.rados.0.smithi071.stdout:146: finishing write tid 4 to smithi07183189-31 2023-12-15T22:56:57.574 INFO:tasks.rados.rados.0.smithi071.stdout:146: finishing write tid 5 to smithi07183189-31 2023-12-15T22:56:57.574 INFO:tasks.rados.rados.0.smithi071.stdout:146: oid 31 updating version 0 to 24 2023-12-15T22:56:57.574 INFO:tasks.rados.rados.0.smithi071.stdout:146: oid 31 version 24 is already newer than 22 2023-12-15T22:56:57.574 INFO:tasks.rados.rados.0.smithi071.stdout:146: oid 31 version 24 is already newer than 23 2023-12-15T22:56:57.574 INFO:tasks.rados.rados.0.smithi071.stdout:146: oid 31 updating version 24 to 25 2023-12-15T22:56:57.574 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 25 (ObjNum 94 snap 7 seq_num 94) dirty exists 2023-12-15T22:56:57.574 INFO:tasks.rados.rados.0.smithi071.stdout:146: left oid 31 (ObjNum 94 snap 7 seq_num 94) 2023-12-15T22:56:57.574 INFO:tasks.rados.rados.0.smithi071.stdout:150: finishing write tid 1 to smithi07183189-3 2023-12-15T22:56:57.574 INFO:tasks.rados.rados.0.smithi071.stdout:150: finishing write tid 2 to smithi07183189-3 2023-12-15T22:56:57.574 INFO:tasks.rados.rados.0.smithi071.stdout:150: finishing write tid 3 to smithi07183189-3 2023-12-15T22:56:57.574 INFO:tasks.rados.rados.0.smithi071.stdout:150: finishing write tid 4 to smithi07183189-3 2023-12-15T22:56:57.574 INFO:tasks.rados.rados.0.smithi071.stdout:150: finishing write tid 5 to smithi07183189-3 2023-12-15T22:56:57.574 INFO:tasks.rados.rados.0.smithi071.stdout:150: oid 3 updating version 0 to 23 2023-12-15T22:56:57.575 INFO:tasks.rados.rados.0.smithi071.stdout:150: oid 3 updating version 23 to 25 2023-12-15T22:56:57.575 INFO:tasks.rados.rados.0.smithi071.stdout:150: oid 3 updating version 25 to 26 2023-12-15T22:56:57.575 INFO:tasks.rados.rados.0.smithi071.stdout:150: oid 3 version 26 is already newer than 25 2023-12-15T22:56:57.575 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 26 (ObjNum 96 snap 7 seq_num 96) dirty exists 2023-12-15T22:56:57.575 INFO:tasks.rados.rados.0.smithi071.stdout:150: left oid 3 (ObjNum 96 snap 7 seq_num 96) 2023-12-15T22:56:57.575 INFO:tasks.rados.rados.0.smithi071.stdout:152: finishing write tid 1 to smithi07183189-5 2023-12-15T22:56:57.575 INFO:tasks.rados.rados.0.smithi071.stdout:152: finishing write tid 2 to smithi07183189-5 2023-12-15T22:56:57.575 INFO:tasks.rados.rados.0.smithi071.stdout:152: finishing write tid 3 to smithi07183189-5 2023-12-15T22:56:57.575 INFO:tasks.rados.rados.0.smithi071.stdout:152: finishing write tid 4 to smithi07183189-5 2023-12-15T22:56:57.575 INFO:tasks.rados.rados.0.smithi071.stdout:152: finishing write tid 5 to smithi07183189-5 2023-12-15T22:56:57.575 INFO:tasks.rados.rados.0.smithi071.stdout:152: oid 5 updating version 0 to 62 2023-12-15T22:56:57.575 INFO:tasks.rados.rados.0.smithi071.stdout:152: oid 5 updating version 62 to 63 2023-12-15T22:56:57.575 INFO:tasks.rados.rados.0.smithi071.stdout:152: oid 5 version 63 is already newer than 62 2023-12-15T22:56:57.575 INFO:tasks.rados.rados.0.smithi071.stdout:152: oid 5 version 63 is already newer than 60 2023-12-15T22:56:57.575 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 5 v 63 (ObjNum 98 snap 7 seq_num 98) dirty exists 2023-12-15T22:56:57.575 INFO:tasks.rados.rados.0.smithi071.stdout:152: left oid 5 (ObjNum 98 snap 7 seq_num 98) 2023-12-15T22:56:57.613 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:56:57.720 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:57.718+0000 7ffa88f11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:57.722 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:57.723+0000 7ffa88f11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:58.458 INFO:tasks.rados.rados.0.smithi071.stdout:142: done (6 left) 2023-12-15T22:56:58.458 INFO:tasks.rados.rados.0.smithi071.stdout:146: done (5 left) 2023-12-15T22:56:58.458 INFO:tasks.rados.rados.0.smithi071.stdout:148: done (4 left) 2023-12-15T22:56:58.458 INFO:tasks.rados.rados.0.smithi071.stdout:150: done (3 left) 2023-12-15T22:56:58.459 INFO:tasks.rados.rados.0.smithi071.stdout:152: done (2 left) 2023-12-15T22:56:58.459 INFO:tasks.rados.rados.0.smithi071.stdout:153: done (1 left) 2023-12-15T22:56:58.459 INFO:tasks.rados.rados.0.smithi071.stdout:154: done (0 left) 2023-12-15T22:56:58.459 INFO:tasks.rados.rados.0.smithi071.stdout:155: read oid 33 snap -1 2023-12-15T22:56:58.459 INFO:tasks.rados.rados.0.smithi071.stdout:155: expect (ObjNum 32 snap 0 seq_num 32) 2023-12-15T22:56:58.459 INFO:tasks.rados.rados.0.smithi071.stdout:156: write (excl) oid 26 current snap is 8 2023-12-15T22:56:58.459 INFO:tasks.rados.rados.0.smithi071.stdout:156: seq_num 99 ranges {463893=562240,1593482=658027,2685935=559341} 2023-12-15T22:56:58.464 INFO:tasks.rados.rados.0.smithi071.stdout:156: writing smithi07183189-26 from 463893 to 1026133 tid 1 2023-12-15T22:56:58.471 INFO:tasks.rados.rados.0.smithi071.stdout:156: writing smithi07183189-26 from 1593482 to 2251509 tid 2 2023-12-15T22:56:58.477 INFO:tasks.rados.rados.0.smithi071.stdout:156: writing smithi07183189-26 from 2685935 to 3245276 tid 3 2023-12-15T22:56:58.477 INFO:tasks.rados.rados.0.smithi071.stdout:157: delete oid 5 current snap is 8 2023-12-15T22:56:58.516 INFO:tasks.rados.rados.0.smithi071.stdout:155: expect (ObjNum 32 snap 0 seq_num 32) 2023-12-15T22:56:58.623 INFO:tasks.rados.rados.0.smithi071.stdout:156: finishing write tid 1 to smithi07183189-26 2023-12-15T22:56:58.623 INFO:tasks.rados.rados.0.smithi071.stdout:156: finishing write tid 2 to smithi07183189-26 2023-12-15T22:56:58.623 INFO:tasks.rados.rados.0.smithi071.stdout:156: finishing write tid 3 to smithi07183189-26 2023-12-15T22:56:58.623 INFO:tasks.rados.rados.0.smithi071.stdout:156: finishing write tid 4 to smithi07183189-26 2023-12-15T22:56:58.623 INFO:tasks.rados.rados.0.smithi071.stdout:156: finishing write tid 5 to smithi07183189-26 2023-12-15T22:56:58.623 INFO:tasks.rados.rados.0.smithi071.stdout:156: oid 26 updating version 0 to 28 2023-12-15T22:56:58.623 INFO:tasks.rados.rados.0.smithi071.stdout:156: oid 26 updating version 28 to 30 2023-12-15T22:56:58.623 INFO:tasks.rados.rados.0.smithi071.stdout:156: oid 26 version 30 is already newer than 27 2023-12-15T22:56:58.623 INFO:tasks.rados.rados.0.smithi071.stdout:156: oid 26 version 30 is already newer than 30 2023-12-15T22:56:58.623 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 30 (ObjNum 99 snap 8 seq_num 99) dirty exists 2023-12-15T22:56:58.624 INFO:tasks.rados.rados.0.smithi071.stdout:156: left oid 26 (ObjNum 99 snap 8 seq_num 99) 2023-12-15T22:56:58.624 INFO:tasks.rados.rados.0.smithi071.stdout:155: done (2 left) 2023-12-15T22:56:58.624 INFO:tasks.rados.rados.0.smithi071.stdout:156: done (1 left) 2023-12-15T22:56:58.624 INFO:tasks.rados.rados.0.smithi071.stdout:157: done (0 left) 2023-12-15T22:56:58.624 INFO:tasks.rados.rados.0.smithi071.stdout:158: read oid 45 snap 7 2023-12-15T22:56:58.624 INFO:tasks.rados.rados.0.smithi071.stdout:158: expect (ObjNum 61 snap 2 seq_num 61) 2023-12-15T22:56:58.624 INFO:tasks.rados.rados.0.smithi071.stdout:159: read oid 31 snap -1 2023-12-15T22:56:58.624 INFO:tasks.rados.rados.0.smithi071.stdout:159: expect (ObjNum 94 snap 7 seq_num 94) 2023-12-15T22:56:58.624 INFO:tasks.rados.rados.0.smithi071.stdout:160: snap_create 2023-12-15T22:56:58.631 INFO:tasks.rados.rados.0.smithi071.stdout:158: expect (ObjNum 61 snap 2 seq_num 61) 2023-12-15T22:56:58.791 INFO:tasks.rados.rados.0.smithi071.stdout:159: expect (ObjNum 94 snap 7 seq_num 94) 2023-12-15T22:56:59.459 INFO:tasks.rados.rados.0.smithi071.stdout:158: done (2 left) 2023-12-15T22:56:59.459 INFO:tasks.rados.rados.0.smithi071.stdout:159: done (1 left) 2023-12-15T22:56:59.459 INFO:tasks.rados.rados.0.smithi071.stdout:160: done (0 left) 2023-12-15T22:56:59.459 INFO:tasks.rados.rados.0.smithi071.stdout:161: snap_remove snap 8 2023-12-15T22:56:59.507 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T22:56:59.507 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T22:56:59.507 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T22:56:59.619 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:59.616+0000 7f77b231e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:59.622 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:59.622+0000 7f77b231e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:59.790 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:56:59.797 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:56:59.797 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-12-15T22:56:59.809 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":41,"stamp":"2023-12-15T22:56:59.457517+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":119145527,"num_objects":67,"num_object_clones":18,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":412,"num_read_kb":152568,"num_write":455,"num_write_kb":91132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":426,"ondisk_log_size":426,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":309272,"kb_used_data":194972,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377178088,"statfs":{"total":386547056640,"available":386230362112,"internally_reserved":0,"allocated":199651328,"data_stored":200446334,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10218590,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":27101,"num_write":36,"num_write_kb":11246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.942070"},"pg_stats":[{"pgid":"3.f","version":"17'6","reported_seq":51,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-12-15T22:56:55.449821+0000","last_change":"2023-12-15T22:56:54.492530+0000","last_active":"2023-12-15T22:56:55.449821+0000","last_peered":"2023-12-15T22:56:55.449821+0000","last_clean":"2023-12-15T22:56:55.449821+0000","last_became_active":"2023-12-15T22:56:39.503797+0000","last_became_peered":"2023-12-15T22:56:39.503797+0000","last_unstale":"2023-12-15T22:56:55.449821+0000","last_undegraded":"2023-12-15T22:56:55.449821+0000","last_fullsized":"2023-12-15T22:56:55.449821+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4537824,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":39,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-12-15T22:56:54.536340+0000","last_change":"2023-12-15T22:56:54.541232+0000","last_active":"2023-12-15T22:56:54.536340+0000","last_peered":"2023-12-15T22:56:54.536340+0000","last_clean":"2023-12-15T22:56:54.536340+0000","last_became_active":"2023-12-15T22:56:39.552856+0000","last_became_peered":"2023-12-15T22:56:39.552856+0000","last_unstale":"2023-12-15T22:56:54.536340+0000","last_undegraded":"2023-12-15T22:56:54.536340+0000","last_fullsized":"2023-12-15T22:56:54.536340+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"29'30","reported_seq":85,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-12-15T22:56:54.520135+0000","last_change":"2023-12-15T22:56:54.521505+0000","last_active":"2023-12-15T22:56:54.520135+0000","last_peered":"2023-12-15T22:56:54.520135+0000","last_clean":"2023-12-15T22:56:54.520135+0000","last_became_active":"2023-12-15T22:56:39.495490+0000","last_became_peered":"2023-12-15T22:56:39.495490+0000","last_unstale":"2023-12-15T22:56:54.520135+0000","last_undegraded":"2023-12-15T22:56:54.520135+0000","last_fullsized":"2023-12-15T22:56:54.520135+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15294172,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":45,"num_read_kb":25190,"num_write":33,"num_write_kb":9051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"29'16","reported_seq":79,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-12-15T22:56:55.466814+0000","last_change":"2023-12-15T22:56:54.495220+0000","last_active":"2023-12-15T22:56:55.466814+0000","last_peered":"2023-12-15T22:56:55.466814+0000","last_clean":"2023-12-15T22:56:55.466814+0000","last_became_active":"2023-12-15T22:56:39.579234+0000","last_became_peered":"2023-12-15T22:56:39.579234+0000","last_unstale":"2023-12-15T22:56:55.466814+0000","last_undegraded":"2023-12-15T22:56:55.466814+0000","last_fullsized":"2023-12-15T22:56:55.466814+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8605085,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":27284,"num_write":21,"num_write_kb":5696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"29'19","reported_seq":61,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-12-15T22:56:55.535720+0000","last_change":"2023-12-15T22:56:55.547245+0000","last_active":"2023-12-15T22:56:55.535720+0000","last_peered":"2023-12-15T22:56:55.535720+0000","last_clean":"2023-12-15T22:56:55.535720+0000","last_became_active":"2023-12-15T22:56:39.572176+0000","last_became_peered":"2023-12-15T22:56:39.572176+0000","last_unstale":"2023-12-15T22:56:55.535720+0000","last_undegraded":"2023-12-15T22:56:55.535720+0000","last_fullsized":"2023-12-15T22:56:55.535720+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5495823,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":3685,"num_write":17,"num_write_kb":5025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"29'14","reported_seq":61,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-12-15T22:56:55.473581+0000","last_change":"2023-12-15T22:56:54.486875+0000","last_active":"2023-12-15T22:56:55.473581+0000","last_peered":"2023-12-15T22:56:55.473581+0000","last_clean":"2023-12-15T22:56:55.473581+0000","last_became_active":"2023-12-15T22:56:39.516361+0000","last_became_peered":"2023-12-15T22:56:39.516361+0000","last_unstale":"2023-12-15T22:56:55.473581+0000","last_undegraded":"2023-12-15T22:56:55.473581+0000","last_fullsized":"2023-12-15T22:56:55.473581+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5714314,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":18,"num_write_kb":5066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"26'57","reported_seq":109,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-12-15T22:56:55.475615+0000","last_change":"2023-12-15T22:56:54.490859+0000","last_active":"2023-12-15T22:56:55.475615+0000","last_peered":"2023-12-15T22:56:55.475615+0000","last_clean":"2023-12-15T22:56:55.475615+0000","last_became_active":"2023-12-15T22:56:39.519748+0000","last_became_peered":"2023-12-15T22:56:39.519748+0000","last_unstale":"2023-12-15T22:56:55.475615+0000","last_undegraded":"2023-12-15T22:56:55.475615+0000","last_fullsized":"2023-12-15T22:56:55.475615+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22084134,"num_objects":13,"num_object_clones":4,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":51,"num_read_kb":30184,"num_write":50,"num_write_kb":14949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"23'22","reported_seq":59,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-12-15T22:56:54.494191+0000","last_change":"2023-12-15T22:56:54.502046+0000","last_active":"2023-12-15T22:56:54.494191+0000","last_peered":"2023-12-15T22:56:54.494191+0000","last_clean":"2023-12-15T22:56:54.494191+0000","last_became_active":"2023-12-15T22:56:39.543179+0000","last_became_peered":"2023-12-15T22:56:39.543179+0000","last_unstale":"2023-12-15T22:56:54.494191+0000","last_undegraded":"2023-12-15T22:56:54.494191+0000","last_fullsized":"2023-12-15T22:56:54.494191+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8050454,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":22,"num_write_kb":5303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":168,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-12-15T22:56:55.548750+0000","last_change":"2023-12-15T22:56:32.607017+0000","last_active":"2023-12-15T22:56:55.548750+0000","last_peered":"2023-12-15T22:56:55.548750+0000","last_clean":"2023-12-15T22:56:55.548750+0000","last_became_active":"2023-12-15T22:56:32.602384+0000","last_became_peered":"2023-12-15T22:56:32.602384+0000","last_unstale":"2023-12-15T22:56:55.548750+0000","last_undegraded":"2023-12-15T22:56:55.548750+0000","last_fullsized":"2023-12-15T22:56:55.548750+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_clean_scrub_stamp":"2023-12-15T22:56:31.525572+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":37,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-12-15T22:56:54.483818+0000","last_change":"2023-12-15T22:56:33.627481+0000","last_active":"2023-12-15T22:56:54.483818+0000","last_peered":"2023-12-15T22:56:54.483818+0000","last_clean":"2023-12-15T22:56:54.483818+0000","last_became_active":"2023-12-15T22:56:33.624689+0000","last_became_peered":"2023-12-15T22:56:33.624689+0000","last_unstale":"2023-12-15T22:56:54.483818+0000","last_undegraded":"2023-12-15T22:56:54.483818+0000","last_fullsized":"2023-12-15T22:56:54.483818+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"24'18","reported_seq":63,"reported_epoch":29,"state":"active+clean+snaptrim","last_fresh":"2023-12-15T22:56:54.475672+0000","last_change":"2023-12-15T22:56:54.475672+0000","last_active":"2023-12-15T22:56:54.475672+0000","last_peered":"2023-12-15T22:56:54.475672+0000","last_clean":"2023-12-15T22:56:54.475672+0000","last_became_active":"2023-12-15T22:56:39.529359+0000","last_became_peered":"2023-12-15T22:56:39.529359+0000","last_unstale":"2023-12-15T22:56:54.475672+0000","last_undegraded":"2023-12-15T22:56:54.475672+0000","last_fullsized":"2023-12-15T22:56:54.475672+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6554631,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":24,"num_read_kb":10696,"num_write":20,"num_write_kb":5567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-12-15T22:56:55.450444+0000","last_change":"2023-12-15T22:56:33.655797+0000","last_active":"2023-12-15T22:56:55.450444+0000","last_peered":"2023-12-15T22:56:55.450444+0000","last_clean":"2023-12-15T22:56:55.450444+0000","last_became_active":"2023-12-15T22:56:33.652885+0000","last_became_peered":"2023-12-15T22:56:33.652885+0000","last_unstale":"2023-12-15T22:56:55.450444+0000","last_undegraded":"2023-12-15T22:56:55.450444+0000","last_fullsized":"2023-12-15T22:56:55.450444+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"23'26","reported_seq":77,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-12-15T22:56:55.547298+0000","last_change":"2023-12-15T22:56:54.476491+0000","last_active":"2023-12-15T22:56:55.547298+0000","last_peered":"2023-12-15T22:56:55.547298+0000","last_clean":"2023-12-15T22:56:55.547298+0000","last_became_active":"2023-12-15T22:56:39.543509+0000","last_became_peered":"2023-12-15T22:56:39.543509+0000","last_unstale":"2023-12-15T22:56:55.547298+0000","last_undegraded":"2023-12-15T22:56:55.547298+0000","last_fullsized":"2023-12-15T22:56:55.547298+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7714314,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":31,"num_read_kb":15193,"num_write":25,"num_write_kb":6249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":37,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-12-15T22:56:54.454011+0000","last_change":"2023-12-15T22:56:33.600485+0000","last_active":"2023-12-15T22:56:54.454011+0000","last_peered":"2023-12-15T22:56:54.454011+0000","last_clean":"2023-12-15T22:56:54.454011+0000","last_became_active":"2023-12-15T22:56:33.597320+0000","last_became_peered":"2023-12-15T22:56:33.597320+0000","last_unstale":"2023-12-15T22:56:54.454011+0000","last_undegraded":"2023-12-15T22:56:54.454011+0000","last_fullsized":"2023-12-15T22:56:54.454011+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"23'21","reported_seq":64,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-12-15T22:56:55.527795+0000","last_change":"2023-12-15T22:56:54.469331+0000","last_active":"2023-12-15T22:56:55.527795+0000","last_peered":"2023-12-15T22:56:55.527795+0000","last_clean":"2023-12-15T22:56:55.527795+0000","last_became_active":"2023-12-15T22:56:39.560980+0000","last_became_peered":"2023-12-15T22:56:39.560980+0000","last_unstale":"2023-12-15T22:56:55.527795+0000","last_undegraded":"2023-12-15T22:56:55.527795+0000","last_fullsized":"2023-12-15T22:56:55.527795+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6403401,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":13739,"num_write":17,"num_write_kb":4752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-12-15T22:56:55.466914+0000","last_change":"2023-12-15T22:56:33.648541+0000","last_active":"2023-12-15T22:56:55.466914+0000","last_peered":"2023-12-15T22:56:55.466914+0000","last_clean":"2023-12-15T22:56:55.466914+0000","last_became_active":"2023-12-15T22:56:33.643681+0000","last_became_peered":"2023-12-15T22:56:33.643681+0000","last_unstale":"2023-12-15T22:56:55.466914+0000","last_undegraded":"2023-12-15T22:56:55.466914+0000","last_fullsized":"2023-12-15T22:56:55.466914+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"26'39","reported_seq":81,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-12-15T22:56:54.540831+0000","last_change":"2023-12-15T22:56:54.543550+0000","last_active":"2023-12-15T22:56:54.540831+0000","last_peered":"2023-12-15T22:56:54.540831+0000","last_clean":"2023-12-15T22:56:54.540831+0000","last_became_active":"2023-12-15T22:56:39.506510+0000","last_became_peered":"2023-12-15T22:56:39.506510+0000","last_unstale":"2023-12-15T22:56:54.540831+0000","last_undegraded":"2023-12-15T22:56:54.540831+0000","last_fullsized":"2023-12-15T22:56:54.540831+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12823609,"num_objects":8,"num_object_clones":3,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":25,"num_read_kb":8678,"num_write":35,"num_write_kb":10420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"17'7","reported_seq":46,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-12-15T22:56:54.504990+0000","last_change":"2023-12-15T22:56:54.521420+0000","last_active":"2023-12-15T22:56:54.504990+0000","last_peered":"2023-12-15T22:56:54.504990+0000","last_clean":"2023-12-15T22:56:54.504990+0000","last_became_active":"2023-12-15T22:56:39.511159+0000","last_became_peered":"2023-12-15T22:56:39.511159+0000","last_unstale":"2023-12-15T22:56:54.504990+0000","last_undegraded":"2023-12-15T22:56:54.504990+0000","last_fullsized":"2023-12-15T22:56:54.504990+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2941192,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":9,"num_write_kb":2133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":37,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-12-15T22:56:54.520507+0000","last_change":"2023-12-15T22:56:33.633890+0000","last_active":"2023-12-15T22:56:54.520507+0000","last_peered":"2023-12-15T22:56:54.520507+0000","last_clean":"2023-12-15T22:56:54.520507+0000","last_became_active":"2023-12-15T22:56:33.631134+0000","last_became_peered":"2023-12-15T22:56:33.631134+0000","last_unstale":"2023-12-15T22:56:54.520507+0000","last_undegraded":"2023-12-15T22:56:54.520507+0000","last_fullsized":"2023-12-15T22:56:54.520507+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"27'30","reported_seq":59,"reported_epoch":29,"state":"active+clean+snaptrim","last_fresh":"2023-12-15T22:56:54.528459+0000","last_change":"2023-12-15T22:56:54.528459+0000","last_active":"2023-12-15T22:56:54.528459+0000","last_peered":"2023-12-15T22:56:54.528459+0000","last_clean":"2023-12-15T22:56:54.528459+0000","last_became_active":"2023-12-15T22:56:39.532730+0000","last_became_peered":"2023-12-15T22:56:39.532730+0000","last_unstale":"2023-12-15T22:56:54.528459+0000","last_undegraded":"2023-12-15T22:56:54.528459+0000","last_fullsized":"2023-12-15T22:56:54.528459+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6420208,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":22,"num_write_kb":6960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-12-15T22:56:55.525119+0000","last_change":"2023-12-15T22:56:33.655736+0000","last_active":"2023-12-15T22:56:55.525119+0000","last_peered":"2023-12-15T22:56:55.525119+0000","last_clean":"2023-12-15T22:56:55.525119+0000","last_became_active":"2023-12-15T22:56:33.652917+0000","last_became_peered":"2023-12-15T22:56:33.652917+0000","last_unstale":"2023-12-15T22:56:55.525119+0000","last_undegraded":"2023-12-15T22:56:55.525119+0000","last_fullsized":"2023-12-15T22:56:55.525119+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"29'21","reported_seq":66,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-12-15T22:56:55.524479+0000","last_change":"2023-12-15T22:56:55.540624+0000","last_active":"2023-12-15T22:56:55.524479+0000","last_peered":"2023-12-15T22:56:55.524479+0000","last_clean":"2023-12-15T22:56:55.524479+0000","last_became_active":"2023-12-15T22:56:39.557252+0000","last_became_peered":"2023-12-15T22:56:39.557252+0000","last_unstale":"2023-12-15T22:56:55.524479+0000","last_undegraded":"2023-12-15T22:56:55.524479+0000","last_fullsized":"2023-12-15T22:56:55.524479+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3361367,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":10324,"num_write":17,"num_write_kb":4875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":37,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-12-15T22:56:54.523627+0000","last_change":"2023-12-15T22:56:33.632348+0000","last_active":"2023-12-15T22:56:54.523627+0000","last_peered":"2023-12-15T22:56:54.523627+0000","last_clean":"2023-12-15T22:56:54.523627+0000","last_became_active":"2023-12-15T22:56:33.629022+0000","last_became_peered":"2023-12-15T22:56:33.629022+0000","last_unstale":"2023-12-15T22:56:54.523627+0000","last_undegraded":"2023-12-15T22:56:54.523627+0000","last_fullsized":"2023-12-15T22:56:54.523627+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"17'3","reported_seq":46,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-12-15T22:56:54.536731+0000","last_change":"2023-12-15T22:56:54.542147+0000","last_active":"2023-12-15T22:56:54.536731+0000","last_peered":"2023-12-15T22:56:54.536731+0000","last_clean":"2023-12-15T22:56:54.536731+0000","last_became_active":"2023-12-15T22:56:39.535145+0000","last_became_peered":"2023-12-15T22:56:39.535145+0000","last_unstale":"2023-12-15T22:56:54.536731+0000","last_undegraded":"2023-12-15T22:56:54.536731+0000","last_fullsized":"2023-12-15T22:56:54.536731+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2554631,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":7486,"num_write":4,"num_write_kb":1011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":37,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-12-15T22:56:54.474851+0000","last_change":"2023-12-15T22:56:33.627005+0000","last_active":"2023-12-15T22:56:54.474851+0000","last_peered":"2023-12-15T22:56:54.474851+0000","last_clean":"2023-12-15T22:56:54.474851+0000","last_became_active":"2023-12-15T22:56:33.622181+0000","last_became_peered":"2023-12-15T22:56:33.622181+0000","last_unstale":"2023-12-15T22:56:54.474851+0000","last_undegraded":"2023-12-15T22:56:54.474851+0000","last_fullsized":"2023-12-15T22:56:54.474851+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":118555159,"num_objects":65,"num_object_clones":18,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":309,"num_read_kb":152477,"num_write":318,"num_write_kb":89238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":329,"ondisk_log_size":329,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705690,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":80488,"kb_used_data":50452,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94291352,"statfs":{"total":96636764160,"available":96554344448,"internally_reserved":0,"allocated":51662848,"data_stored":51692038,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705688,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70664,"kb_used_data":43840,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94301176,"statfs":{"total":96636764160,"available":96564404224,"internally_reserved":0,"allocated":44892160,"data_stored":46065371,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705688,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":72112,"kb_used_data":45288,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94299728,"statfs":{"total":96636764160,"available":96562921472,"internally_reserved":0,"allocated":46374912,"data_stored":45949608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705690,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":86008,"kb_used_data":55392,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94285832,"statfs":{"total":96636764160,"available":96548691968,"internally_reserved":0,"allocated":56721408,"data_stored":56739317,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-15T22:56:59.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:59.892+0000 7fa98b39e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:56:59.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:56:59.896+0000 7fa98b39e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:00.460 INFO:tasks.rados.rados.0.smithi071.stdout:161: done (0 left) 2023-12-15T22:57:00.460 INFO:tasks.rados.rados.0.smithi071.stdout:162: write (excl) oid 26 current snap is 9 2023-12-15T22:57:00.460 INFO:tasks.rados.rados.0.smithi071.stdout:162: seq_num 101 ranges {497507=712738,1710446=661696,3140781=556726} 2023-12-15T22:57:00.466 INFO:tasks.rados.rados.0.smithi071.stdout:162: writing smithi07183189-26 from 497507 to 1210245 tid 1 2023-12-15T22:57:00.473 INFO:tasks.rados.rados.0.smithi071.stdout:162: writing smithi07183189-26 from 1710446 to 2372142 tid 2 2023-12-15T22:57:00.480 INFO:tasks.rados.rados.0.smithi071.stdout:162: writing smithi07183189-26 from 3140781 to 3697507 tid 3 2023-12-15T22:57:00.480 INFO:tasks.rados.rados.0.smithi071.stdout:163: snap_remove snap 7 2023-12-15T22:57:01.467 INFO:tasks.rados.rados.0.smithi071.stdout:163: done (1 left) 2023-12-15T22:57:01.467 INFO:tasks.rados.rados.0.smithi071.stdout:164: read oid 7 snap -1 2023-12-15T22:57:01.467 INFO:tasks.rados.rados.0.smithi071.stdout:164: expect (ObjNum 67 snap 3 seq_num 67) 2023-12-15T22:57:01.467 INFO:tasks.rados.rados.0.smithi071.stdout:165: delete oid 12 current snap is 9 2023-12-15T22:57:01.467 INFO:tasks.rados.rados.0.smithi071.stdout:162: finishing write tid 1 to smithi07183189-26 2023-12-15T22:57:01.467 INFO:tasks.rados.rados.0.smithi071.stdout:162: finishing write tid 2 to smithi07183189-26 2023-12-15T22:57:01.468 INFO:tasks.rados.rados.0.smithi071.stdout:162: finishing write tid 3 to smithi07183189-26 2023-12-15T22:57:01.468 INFO:tasks.rados.rados.0.smithi071.stdout:162: finishing write tid 4 to smithi07183189-26 2023-12-15T22:57:01.468 INFO:tasks.rados.rados.0.smithi071.stdout:162: finishing write tid 5 to smithi07183189-26 2023-12-15T22:57:01.468 INFO:tasks.rados.rados.0.smithi071.stdout:162: oid 26 updating version 0 to 34 2023-12-15T22:57:01.468 INFO:tasks.rados.rados.0.smithi071.stdout:162: oid 26 version 34 is already newer than 34 2023-12-15T22:57:01.468 INFO:tasks.rados.rados.0.smithi071.stdout:162: oid 26 version 34 is already newer than 32 2023-12-15T22:57:01.468 INFO:tasks.rados.rados.0.smithi071.stdout:162: oid 26 version 34 is already newer than 31 2023-12-15T22:57:01.468 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 34 (ObjNum 101 snap 9 seq_num 101) dirty exists 2023-12-15T22:57:01.468 INFO:tasks.rados.rados.0.smithi071.stdout:162: left oid 26 (ObjNum 101 snap 9 seq_num 101) 2023-12-15T22:57:01.529 INFO:tasks.rados.rados.0.smithi071.stdout:162: done (2 left) 2023-12-15T22:57:01.529 INFO:tasks.rados.rados.0.smithi071.stdout:165: done (1 left) 2023-12-15T22:57:01.530 INFO:tasks.rados.rados.0.smithi071.stdout:166: delete oid 10 current snap is 9 2023-12-15T22:57:01.534 INFO:tasks.rados.rados.0.smithi071.stdout:164: expect (ObjNum 67 snap 3 seq_num 67) 2023-12-15T22:57:01.697 INFO:tasks.rados.rados.0.smithi071.stdout:164: done (1 left) 2023-12-15T22:57:01.697 INFO:tasks.rados.rados.0.smithi071.stdout:166: done (0 left) 2023-12-15T22:57:01.697 INFO:tasks.rados.rados.0.smithi071.stdout:167: delete oid 11 current snap is 9 2023-12-15T22:57:01.710 INFO:tasks.rados.rados.0.smithi071.stdout:167: done (0 left) 2023-12-15T22:57:01.710 INFO:tasks.rados.rados.0.smithi071.stdout:168: write (excl) oid 44 current snap is 9 2023-12-15T22:57:01.710 INFO:tasks.rados.rados.0.smithi071.stdout:168: seq_num 105 ranges {564735=613734,1828021=669034,3134120=630615} 2023-12-15T22:57:01.717 INFO:tasks.rados.rados.0.smithi071.stdout:168: writing smithi07183189-44 from 564735 to 1178469 tid 1 2023-12-15T22:57:01.723 INFO:tasks.rados.rados.0.smithi071.stdout:168: writing smithi07183189-44 from 1828021 to 2497055 tid 2 2023-12-15T22:57:01.732 INFO:tasks.rados.rados.0.smithi071.stdout:168: writing smithi07183189-44 from 3134120 to 3764735 tid 3 2023-12-15T22:57:01.732 INFO:tasks.rados.rados.0.smithi071.stdout:169: write oid 2 current snap is 9 2023-12-15T22:57:01.732 INFO:tasks.rados.rados.0.smithi071.stdout:169: seq_num 106 ranges {581542=688983,1886503=412692,2761543=670951} 2023-12-15T22:57:01.739 INFO:tasks.rados.rados.0.smithi071.stdout:169: writing smithi07183189-2 from 581542 to 1270525 tid 1 2023-12-15T22:57:01.745 INFO:tasks.rados.rados.0.smithi071.stdout:169: writing smithi07183189-2 from 1886503 to 2299195 tid 2 2023-12-15T22:57:01.752 INFO:tasks.rados.rados.0.smithi071.stdout:169: writing smithi07183189-2 from 2761543 to 3432494 tid 3 2023-12-15T22:57:01.752 INFO:tasks.rados.rados.0.smithi071.stdout:170: read oid 21 snap -1 2023-12-15T22:57:01.752 INFO:tasks.rados.rados.0.smithi071.stdout:170: expect (ObjNum 55 snap 0 seq_num 55) 2023-12-15T22:57:01.752 INFO:tasks.rados.rados.0.smithi071.stdout:171: write oid 17 current snap is 9 2023-12-15T22:57:01.752 INFO:tasks.rados.rados.0.smithi071.stdout:168: finishing write tid 1 to smithi07183189-44 2023-12-15T22:57:01.752 INFO:tasks.rados.rados.0.smithi071.stdout:171: seq_num 107 ranges {598349=480585,1544985=672703,2905319=482223} 2023-12-15T22:57:01.758 INFO:tasks.rados.rados.0.smithi071.stdout:171: writing smithi07183189-17 from 598349 to 1078934 tid 1 2023-12-15T22:57:01.764 INFO:tasks.rados.rados.0.smithi071.stdout:171: writing smithi07183189-17 from 1544985 to 2217688 tid 2 2023-12-15T22:57:01.771 INFO:tasks.rados.rados.0.smithi071.stdout:171: writing smithi07183189-17 from 2905319 to 3387542 tid 3 2023-12-15T22:57:01.771 INFO:tasks.rados.rados.0.smithi071.stdout:172: write (excl) oid 15 current snap is 9 2023-12-15T22:57:01.771 INFO:tasks.rados.rados.0.smithi071.stdout:172: seq_num 108 ranges {615156=555834,1603467=416361,2816389=693495} 2023-12-15T22:57:01.777 INFO:tasks.rados.rados.0.smithi071.stdout:172: writing smithi07183189-15 from 615156 to 1170990 tid 1 2023-12-15T22:57:01.783 INFO:tasks.rados.rados.0.smithi071.stdout:172: writing smithi07183189-15 from 1603467 to 2019828 tid 2 2023-12-15T22:57:01.792 INFO:tasks.rados.rados.0.smithi071.stdout:172: writing smithi07183189-15 from 2816389 to 3509884 tid 3 2023-12-15T22:57:01.792 INFO:tasks.rados.rados.0.smithi071.stdout:173: read oid 9 snap -1 2023-12-15T22:57:01.792 INFO:tasks.rados.rados.0.smithi071.stdout:173: expect (ObjNum 91 snap 7 seq_num 91) 2023-12-15T22:57:01.792 INFO:tasks.rados.rados.0.smithi071.stdout:174: write (excl) oid 39 current snap is 9 2023-12-15T22:57:01.792 INFO:tasks.rados.rados.0.smithi071.stdout:174: seq_num 109 ranges {631963=631083,2061949=560019,3243812=504767} 2023-12-15T22:57:01.799 INFO:tasks.rados.rados.0.smithi071.stdout:174: writing smithi07183189-39 from 631963 to 1263046 tid 1 2023-12-15T22:57:01.807 INFO:tasks.rados.rados.0.smithi071.stdout:174: writing smithi07183189-39 from 2061949 to 2621968 tid 2 2023-12-15T22:57:01.815 INFO:tasks.rados.rados.0.smithi071.stdout:174: writing smithi07183189-39 from 3243812 to 3748579 tid 3 2023-12-15T22:57:01.815 INFO:tasks.rados.rados.0.smithi071.stdout:175: read oid 3 snap -1 2023-12-15T22:57:01.815 INFO:tasks.rados.rados.0.smithi071.stdout:175: expect (ObjNum 96 snap 7 seq_num 96) 2023-12-15T22:57:01.815 INFO:tasks.rados.rados.0.smithi071.stdout:176: write oid 35 current snap is 9 2023-12-15T22:57:01.815 INFO:tasks.rados.rados.0.smithi071.stdout:176: seq_num 110 ranges {648770=706332,2120431=420030,3271235=432392} 2023-12-15T22:57:01.823 INFO:tasks.rados.rados.0.smithi071.stdout:176: writing smithi07183189-35 from 648770 to 1355102 tid 1 2023-12-15T22:57:01.830 INFO:tasks.rados.rados.0.smithi071.stdout:176: writing smithi07183189-35 from 2120431 to 2540461 tid 2 2023-12-15T22:57:01.837 INFO:tasks.rados.rados.0.smithi071.stdout:176: writing smithi07183189-35 from 3271235 to 3703627 tid 3 2023-12-15T22:57:01.837 INFO:tasks.rados.rados.0.smithi071.stdout:177: snap_create 2023-12-15T22:57:01.837 INFO:tasks.rados.rados.0.smithi071.stdout:170: expect (ObjNum 55 snap 0 seq_num 55) 2023-12-15T22:57:01.965 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:57:02.006 INFO:tasks.rados.rados.0.smithi071.stdout:175: expect (ObjNum 96 snap 7 seq_num 96) 2023-12-15T22:57:02.076 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:02.074+0000 7f3b1795e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:02.079 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:02.079+0000 7f3b1795e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:02.191 INFO:tasks.rados.rados.0.smithi071.stdout:168: finishing write tid 2 to smithi07183189-44 2023-12-15T22:57:02.191 INFO:tasks.rados.rados.0.smithi071.stdout:171: finishing write tid 1 to smithi07183189-17 2023-12-15T22:57:02.191 INFO:tasks.rados.rados.0.smithi071.stdout:174: finishing write tid 1 to smithi07183189-39 2023-12-15T22:57:02.191 INFO:tasks.rados.rados.0.smithi071.stdout:172: finishing write tid 1 to smithi07183189-15 2023-12-15T22:57:02.191 INFO:tasks.rados.rados.0.smithi071.stdout:169: finishing write tid 1 to smithi07183189-2 2023-12-15T22:57:02.191 INFO:tasks.rados.rados.0.smithi071.stdout:168: finishing write tid 3 to smithi07183189-44 2023-12-15T22:57:02.191 INFO:tasks.rados.rados.0.smithi071.stdout:171: finishing write tid 2 to smithi07183189-17 2023-12-15T22:57:02.191 INFO:tasks.rados.rados.0.smithi071.stdout:173: expect (ObjNum 91 snap 7 seq_num 91) 2023-12-15T22:57:02.413 INFO:tasks.rados.rados.0.smithi071.stdout:174: finishing write tid 2 to smithi07183189-39 2023-12-15T22:57:02.413 INFO:tasks.rados.rados.0.smithi071.stdout:172: finishing write tid 2 to smithi07183189-15 2023-12-15T22:57:02.413 INFO:tasks.rados.rados.0.smithi071.stdout:174: finishing write tid 3 to smithi07183189-39 2023-12-15T22:57:02.413 INFO:tasks.rados.rados.0.smithi071.stdout:172: finishing write tid 3 to smithi07183189-15 2023-12-15T22:57:02.413 INFO:tasks.rados.rados.0.smithi071.stdout:169: finishing write tid 2 to smithi07183189-2 2023-12-15T22:57:02.413 INFO:tasks.rados.rados.0.smithi071.stdout:171: finishing write tid 3 to smithi07183189-17 2023-12-15T22:57:02.413 INFO:tasks.rados.rados.0.smithi071.stdout:176: finishing write tid 1 to smithi07183189-35 2023-12-15T22:57:02.413 INFO:tasks.rados.rados.0.smithi071.stdout:168: finishing write tid 4 to smithi07183189-44 2023-12-15T22:57:02.413 INFO:tasks.rados.rados.0.smithi071.stdout:168: finishing write tid 5 to smithi07183189-44 2023-12-15T22:57:02.413 INFO:tasks.rados.rados.0.smithi071.stdout:168: oid 44 updating version 0 to 36 2023-12-15T22:57:02.413 INFO:tasks.rados.rados.0.smithi071.stdout:168: oid 44 version 36 is already newer than 34 2023-12-15T22:57:02.414 INFO:tasks.rados.rados.0.smithi071.stdout:168: oid 44 version 36 is already newer than 33 2023-12-15T22:57:02.414 INFO:tasks.rados.rados.0.smithi071.stdout:168: oid 44 version 36 is already newer than 35 2023-12-15T22:57:02.414 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 36 (ObjNum 105 snap 9 seq_num 105) dirty exists 2023-12-15T22:57:02.414 INFO:tasks.rados.rados.0.smithi071.stdout:168: left oid 44 (ObjNum 105 snap 9 seq_num 105) 2023-12-15T22:57:02.414 INFO:tasks.rados.rados.0.smithi071.stdout:172: finishing write tid 4 to smithi07183189-15 2023-12-15T22:57:02.414 INFO:tasks.rados.rados.0.smithi071.stdout:172: finishing write tid 5 to smithi07183189-15 2023-12-15T22:57:02.414 INFO:tasks.rados.rados.0.smithi071.stdout:172: oid 15 updating version 0 to 38 2023-12-15T22:57:02.414 INFO:tasks.rados.rados.0.smithi071.stdout:172: oid 15 version 38 is already newer than 37 2023-12-15T22:57:02.414 INFO:tasks.rados.rados.0.smithi071.stdout:172: oid 15 version 38 is already newer than 37 2023-12-15T22:57:02.414 INFO:tasks.rados.rados.0.smithi071.stdout:172: oid 15 version 38 is already newer than 35 2023-12-15T22:57:02.414 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 38 (ObjNum 108 snap 9 seq_num 108) dirty exists 2023-12-15T22:57:02.414 INFO:tasks.rados.rados.0.smithi071.stdout:172: left oid 15 (ObjNum 108 snap 9 seq_num 108) 2023-12-15T22:57:02.414 INFO:tasks.rados.rados.0.smithi071.stdout:169: finishing write tid 3 to smithi07183189-2 2023-12-15T22:57:02.414 INFO:tasks.rados.rados.0.smithi071.stdout:174: finishing write tid 4 to smithi07183189-39 2023-12-15T22:57:02.414 INFO:tasks.rados.rados.0.smithi071.stdout:174: finishing write tid 5 to smithi07183189-39 2023-12-15T22:57:02.415 INFO:tasks.rados.rados.0.smithi071.stdout:174: oid 39 updating version 0 to 43 2023-12-15T22:57:02.415 INFO:tasks.rados.rados.0.smithi071.stdout:174: oid 39 version 43 is already newer than 40 2023-12-15T22:57:02.415 INFO:tasks.rados.rados.0.smithi071.stdout:174: oid 39 version 43 is already newer than 42 2023-12-15T22:57:02.415 INFO:tasks.rados.rados.0.smithi071.stdout:174: oid 39 version 43 is already newer than 42 2023-12-15T22:57:02.415 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 39 v 43 (ObjNum 109 snap 9 seq_num 109) dirty exists 2023-12-15T22:57:02.415 INFO:tasks.rados.rados.0.smithi071.stdout:174: left oid 39 (ObjNum 109 snap 9 seq_num 109) 2023-12-15T22:57:02.415 INFO:tasks.rados.rados.0.smithi071.stdout:171: finishing write tid 4 to smithi07183189-17 2023-12-15T22:57:02.415 INFO:tasks.rados.rados.0.smithi071.stdout:171: finishing write tid 5 to smithi07183189-17 2023-12-15T22:57:02.415 INFO:tasks.rados.rados.0.smithi071.stdout:171: oid 17 updating version 0 to 47 2023-12-15T22:57:02.415 INFO:tasks.rados.rados.0.smithi071.stdout:171: oid 17 updating version 47 to 48 2023-12-15T22:57:02.415 INFO:tasks.rados.rados.0.smithi071.stdout:171: oid 17 version 48 is already newer than 46 2023-12-15T22:57:02.415 INFO:tasks.rados.rados.0.smithi071.stdout:171: oid 17 updating version 48 to 49 2023-12-15T22:57:02.415 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 17 v 49 (ObjNum 107 snap 9 seq_num 107) dirty exists 2023-12-15T22:57:02.415 INFO:tasks.rados.rados.0.smithi071.stdout:171: left oid 17 (ObjNum 107 snap 9 seq_num 107) 2023-12-15T22:57:02.415 INFO:tasks.rados.rados.0.smithi071.stdout:176: finishing write tid 2 to smithi07183189-35 2023-12-15T22:57:02.415 INFO:tasks.rados.rados.0.smithi071.stdout:169: finishing write tid 4 to smithi07183189-2 2023-12-15T22:57:02.416 INFO:tasks.rados.rados.0.smithi071.stdout:169: finishing write tid 5 to smithi07183189-2 2023-12-15T22:57:02.416 INFO:tasks.rados.rados.0.smithi071.stdout:169: oid 2 updating version 0 to 15 2023-12-15T22:57:02.416 INFO:tasks.rados.rados.0.smithi071.stdout:169: oid 2 updating version 15 to 18 2023-12-15T22:57:02.416 INFO:tasks.rados.rados.0.smithi071.stdout:169: oid 2 version 18 is already newer than 16 2023-12-15T22:57:02.416 INFO:tasks.rados.rados.0.smithi071.stdout:169: oid 2 version 18 is already newer than 17 2023-12-15T22:57:02.416 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 18 (ObjNum 106 snap 9 seq_num 106) dirty exists 2023-12-15T22:57:02.416 INFO:tasks.rados.rados.0.smithi071.stdout:169: left oid 2 (ObjNum 106 snap 9 seq_num 106) 2023-12-15T22:57:02.416 INFO:tasks.rados.rados.0.smithi071.stdout:176: finishing write tid 3 to smithi07183189-35 2023-12-15T22:57:02.416 INFO:tasks.rados.rados.0.smithi071.stdout:176: finishing write tid 4 to smithi07183189-35 2023-12-15T22:57:02.416 INFO:tasks.rados.rados.0.smithi071.stdout:176: finishing write tid 5 to smithi07183189-35 2023-12-15T22:57:02.416 INFO:tasks.rados.rados.0.smithi071.stdout:176: oid 35 updating version 0 to 22 2023-12-15T22:57:02.416 INFO:tasks.rados.rados.0.smithi071.stdout:176: oid 35 updating version 22 to 23 2023-12-15T22:57:02.416 INFO:tasks.rados.rados.0.smithi071.stdout:176: oid 35 updating version 23 to 24 2023-12-15T22:57:02.416 INFO:tasks.rados.rados.0.smithi071.stdout:176: oid 35 version 24 is already newer than 21 2023-12-15T22:57:02.416 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 24 (ObjNum 110 snap 9 seq_num 110) dirty exists 2023-12-15T22:57:02.417 INFO:tasks.rados.rados.0.smithi071.stdout:176: left oid 35 (ObjNum 110 snap 9 seq_num 110) 2023-12-15T22:57:02.469 INFO:tasks.rados.rados.0.smithi071.stdout:168: done (9 left) 2023-12-15T22:57:02.469 INFO:tasks.rados.rados.0.smithi071.stdout:169: done (8 left) 2023-12-15T22:57:02.469 INFO:tasks.rados.rados.0.smithi071.stdout:170: done (7 left) 2023-12-15T22:57:02.469 INFO:tasks.rados.rados.0.smithi071.stdout:171: done (6 left) 2023-12-15T22:57:02.469 INFO:tasks.rados.rados.0.smithi071.stdout:172: done (5 left) 2023-12-15T22:57:02.470 INFO:tasks.rados.rados.0.smithi071.stdout:173: done (4 left) 2023-12-15T22:57:02.470 INFO:tasks.rados.rados.0.smithi071.stdout:174: done (3 left) 2023-12-15T22:57:02.470 INFO:tasks.rados.rados.0.smithi071.stdout:175: done (2 left) 2023-12-15T22:57:02.470 INFO:tasks.rados.rados.0.smithi071.stdout:176: done (1 left) 2023-12-15T22:57:02.470 INFO:tasks.rados.rados.0.smithi071.stdout:177: done (0 left) 2023-12-15T22:57:02.470 INFO:tasks.rados.rados.0.smithi071.stdout:178: delete oid 23 current snap is 10 2023-12-15T22:57:02.489 INFO:tasks.rados.rados.0.smithi071.stdout:178: done (0 left) 2023-12-15T22:57:02.490 INFO:tasks.rados.rados.0.smithi071.stdout:179: delete oid 36 current snap is 10 2023-12-15T22:57:02.533 INFO:tasks.rados.rados.0.smithi071.stdout:179: done (0 left) 2023-12-15T22:57:02.533 INFO:tasks.rados.rados.0.smithi071.stdout:180: write oid 41 current snap is 10 2023-12-15T22:57:02.533 INFO:tasks.rados.rados.0.smithi071.stdout:180: seq_num 113 ranges {699191=532079,1779524=567357,2837151=666208} 2023-12-15T22:57:02.539 INFO:tasks.rados.rados.0.smithi071.stdout:180: writing smithi07183189-41 from 699191 to 1231270 tid 1 2023-12-15T22:57:02.546 INFO:tasks.rados.rados.0.smithi071.stdout:180: writing smithi07183189-41 from 1779524 to 2346881 tid 2 2023-12-15T22:57:02.552 INFO:tasks.rados.rados.0.smithi071.stdout:180: writing smithi07183189-41 from 2837151 to 3503359 tid 3 2023-12-15T22:57:02.552 INFO:tasks.rados.rados.0.smithi071.stdout:181: write oid 20 current snap is 10 2023-12-15T22:57:02.552 INFO:tasks.rados.rados.0.smithi071.stdout:181: seq_num 114 ranges {715998=607328,1838006=427368,2864574=477480,3863835=52163} 2023-12-15T22:57:02.559 INFO:tasks.rados.rados.0.smithi071.stdout:181: writing smithi07183189-20 from 715998 to 1323326 tid 1 2023-12-15T22:57:02.565 INFO:tasks.rados.rados.0.smithi071.stdout:181: writing smithi07183189-20 from 1838006 to 2265374 tid 2 2023-12-15T22:57:02.571 INFO:tasks.rados.rados.0.smithi071.stdout:181: writing smithi07183189-20 from 2864574 to 3342054 tid 3 2023-12-15T22:57:02.574 INFO:tasks.rados.rados.0.smithi071.stdout:181: writing smithi07183189-20 from 3863835 to 3915998 tid 4 2023-12-15T22:57:02.574 INFO:tasks.rados.rados.0.smithi071.stdout:182: snap_remove snap 9 2023-12-15T22:57:03.474 INFO:tasks.rados.rados.0.smithi071.stdout:182: done (2 left) 2023-12-15T22:57:03.474 INFO:tasks.rados.rados.0.smithi071.stdout:183: delete oid 30 current snap is 10 2023-12-15T22:57:03.474 INFO:tasks.rados.rados.0.smithi071.stdout:180: finishing write tid 1 to smithi07183189-41 2023-12-15T22:57:03.474 INFO:tasks.rados.rados.0.smithi071.stdout:181: finishing write tid 1 to smithi07183189-20 2023-12-15T22:57:03.474 INFO:tasks.rados.rados.0.smithi071.stdout:180: finishing write tid 2 to smithi07183189-41 2023-12-15T22:57:03.474 INFO:tasks.rados.rados.0.smithi071.stdout:181: finishing write tid 2 to smithi07183189-20 2023-12-15T22:57:03.474 INFO:tasks.rados.rados.0.smithi071.stdout:180: finishing write tid 3 to smithi07183189-41 2023-12-15T22:57:03.474 INFO:tasks.rados.rados.0.smithi071.stdout:181: finishing write tid 3 to smithi07183189-20 2023-12-15T22:57:03.474 INFO:tasks.rados.rados.0.smithi071.stdout:180: finishing write tid 4 to smithi07183189-41 2023-12-15T22:57:03.474 INFO:tasks.rados.rados.0.smithi071.stdout:180: finishing write tid 5 to smithi07183189-41 2023-12-15T22:57:03.474 INFO:tasks.rados.rados.0.smithi071.stdout:180: oid 41 updating version 0 to 32 2023-12-15T22:57:03.474 INFO:tasks.rados.rados.0.smithi071.stdout:180: oid 41 updating version 32 to 33 2023-12-15T22:57:03.475 INFO:tasks.rados.rados.0.smithi071.stdout:180: oid 41 updating version 33 to 35 2023-12-15T22:57:03.475 INFO:tasks.rados.rados.0.smithi071.stdout:180: oid 41 version 35 is already newer than 34 2023-12-15T22:57:03.475 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 41 v 35 (ObjNum 113 snap 10 seq_num 113) dirty exists 2023-12-15T22:57:03.475 INFO:tasks.rados.rados.0.smithi071.stdout:180: left oid 41 (ObjNum 113 snap 10 seq_num 113) 2023-12-15T22:57:03.475 INFO:tasks.rados.rados.0.smithi071.stdout:181: finishing write tid 4 to smithi07183189-20 2023-12-15T22:57:03.475 INFO:tasks.rados.rados.0.smithi071.stdout:181: finishing write tid 5 to smithi07183189-20 2023-12-15T22:57:03.475 INFO:tasks.rados.rados.0.smithi071.stdout:181: finishing write tid 6 to smithi07183189-20 2023-12-15T22:57:03.475 INFO:tasks.rados.rados.0.smithi071.stdout:181: oid 20 updating version 0 to 73 2023-12-15T22:57:03.475 INFO:tasks.rados.rados.0.smithi071.stdout:181: oid 20 version 73 is already newer than 70 2023-12-15T22:57:03.475 INFO:tasks.rados.rados.0.smithi071.stdout:181: oid 20 version 73 is already newer than 69 2023-12-15T22:57:03.475 INFO:tasks.rados.rados.0.smithi071.stdout:181: oid 20 version 73 is already newer than 72 2023-12-15T22:57:03.475 INFO:tasks.rados.rados.0.smithi071.stdout:181: oid 20 version 73 is already newer than 71 2023-12-15T22:57:03.475 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 20 v 73 (ObjNum 114 snap 10 seq_num 114) dirty exists 2023-12-15T22:57:03.475 INFO:tasks.rados.rados.0.smithi071.stdout:181: left oid 20 (ObjNum 114 snap 10 seq_num 114) 2023-12-15T22:57:03.532 INFO:tasks.rados.rados.0.smithi071.stdout:180: done (2 left) 2023-12-15T22:57:03.532 INFO:tasks.rados.rados.0.smithi071.stdout:181: done (1 left) 2023-12-15T22:57:03.532 INFO:tasks.rados.rados.0.smithi071.stdout:183: done (0 left) 2023-12-15T22:57:03.532 INFO:tasks.rados.rados.0.smithi071.stdout:184: write (excl) oid 45 current snap is 10 2023-12-15T22:57:03.532 INFO:tasks.rados.rados.0.smithi071.stdout:184: seq_num 116 ranges {749612=474179,1954970=431037,2919420=616377} 2023-12-15T22:57:03.538 INFO:tasks.rados.rados.0.smithi071.stdout:184: writing smithi07183189-45 from 749612 to 1223791 tid 1 2023-12-15T22:57:03.544 INFO:tasks.rados.rados.0.smithi071.stdout:184: writing smithi07183189-45 from 1954970 to 2386007 tid 2 2023-12-15T22:57:03.551 INFO:tasks.rados.rados.0.smithi071.stdout:184: writing smithi07183189-45 from 2919420 to 3535797 tid 3 2023-12-15T22:57:03.551 INFO:tasks.rados.rados.0.smithi071.stdout:185: write (excl) oid 4 current snap is 10 2023-12-15T22:57:03.551 INFO:tasks.rados.rados.0.smithi071.stdout:185: seq_num 117 ranges {766419=549428,2013452=574695,3346843=427649} 2023-12-15T22:57:03.558 INFO:tasks.rados.rados.0.smithi071.stdout:185: writing smithi07183189-4 from 766419 to 1315847 tid 1 2023-12-15T22:57:03.566 INFO:tasks.rados.rados.0.smithi071.stdout:185: writing smithi07183189-4 from 2013452 to 2588147 tid 2 2023-12-15T22:57:03.573 INFO:tasks.rados.rados.0.smithi071.stdout:185: writing smithi07183189-4 from 3346843 to 3774492 tid 3 2023-12-15T22:57:03.573 INFO:tasks.rados.rados.0.smithi071.stdout:186: delete oid 30 current snap is 10 2023-12-15T22:57:03.583 INFO:tasks.rados.rados.0.smithi071.stdout:186: done (2 left) 2023-12-15T22:57:03.584 INFO:tasks.rados.rados.0.smithi071.stdout:187: write (excl) oid 3 current snap is 10 2023-12-15T22:57:03.584 INFO:tasks.rados.rados.0.smithi071.stdout:187: seq_num 119 ranges {400033=699926,1614063=385970} 2023-12-15T22:57:03.590 INFO:tasks.rados.rados.0.smithi071.stdout:187: writing smithi07183189-3 from 400033 to 1099959 tid 1 2023-12-15T22:57:03.595 INFO:tasks.rados.rados.0.smithi071.stdout:187: writing smithi07183189-3 from 1614063 to 2000033 tid 2 2023-12-15T22:57:03.595 INFO:tasks.rados.rados.0.smithi071.stdout:188: write oid 26 current snap is 10 2023-12-15T22:57:03.596 INFO:tasks.rados.rados.0.smithi071.stdout:188: seq_num 120 ranges {416840=775175,1672545=344295} 2023-12-15T22:57:03.602 INFO:tasks.rados.rados.0.smithi071.stdout:188: writing smithi07183189-26 from 416840 to 1192015 tid 1 2023-12-15T22:57:03.607 INFO:tasks.rados.rados.0.smithi071.stdout:188: writing smithi07183189-26 from 1672545 to 2016840 tid 2 2023-12-15T22:57:03.607 INFO:tasks.rados.rados.0.smithi071.stdout:189: snap_create 2023-12-15T22:57:03.664 INFO:tasks.rados.rados.0.smithi071.stdout:185: finishing write tid 1 to smithi07183189-4 2023-12-15T22:57:03.665 INFO:tasks.rados.rados.0.smithi071.stdout:184: finishing write tid 1 to smithi07183189-45 2023-12-15T22:57:03.688 INFO:tasks.rados.rados.0.smithi071.stdout:188: finishing write tid 1 to smithi07183189-26 2023-12-15T22:57:03.690 INFO:tasks.rados.rados.0.smithi071.stdout:187: finishing write tid 1 to smithi07183189-3 2023-12-15T22:57:03.702 INFO:tasks.rados.rados.0.smithi071.stdout:184: finishing write tid 2 to smithi07183189-45 2023-12-15T22:57:03.708 INFO:tasks.rados.rados.0.smithi071.stdout:185: finishing write tid 2 to smithi07183189-4 2023-12-15T22:57:03.728 INFO:tasks.rados.rados.0.smithi071.stdout:187: finishing write tid 2 to smithi07183189-3 2023-12-15T22:57:03.736 INFO:tasks.rados.rados.0.smithi071.stdout:185: finishing write tid 3 to smithi07183189-4 2023-12-15T22:57:03.739 INFO:tasks.rados.rados.0.smithi071.stdout:184: finishing write tid 3 to smithi07183189-45 2023-12-15T22:57:03.749 INFO:tasks.rados.rados.0.smithi071.stdout:188: finishing write tid 2 to smithi07183189-26 2023-12-15T22:57:03.749 INFO:tasks.rados.rados.0.smithi071.stdout:184: finishing write tid 4 to smithi07183189-45 2023-12-15T22:57:03.751 INFO:tasks.rados.rados.0.smithi071.stdout:184: finishing write tid 5 to smithi07183189-45 2023-12-15T22:57:03.751 INFO:tasks.rados.rados.0.smithi071.stdout:184: oid 45 updating version 0 to 78 2023-12-15T22:57:03.751 INFO:tasks.rados.rados.0.smithi071.stdout:184: oid 45 version 78 is already newer than 76 2023-12-15T22:57:03.751 INFO:tasks.rados.rados.0.smithi071.stdout:184: oid 45 updating version 78 to 79 2023-12-15T22:57:03.751 INFO:tasks.rados.rados.0.smithi071.stdout:184: oid 45 version 79 is already newer than 77 2023-12-15T22:57:03.751 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 79 (ObjNum 116 snap 10 seq_num 116) dirty exists 2023-12-15T22:57:03.751 INFO:tasks.rados.rados.0.smithi071.stdout:184: left oid 45 (ObjNum 116 snap 10 seq_num 116) 2023-12-15T22:57:03.755 INFO:tasks.rados.rados.0.smithi071.stdout:187: finishing write tid 3 to smithi07183189-3 2023-12-15T22:57:03.760 INFO:tasks.rados.rados.0.smithi071.stdout:187: finishing write tid 4 to smithi07183189-3 2023-12-15T22:57:03.760 INFO:tasks.rados.rados.0.smithi071.stdout:187: oid 3 updating version 0 to 28 2023-12-15T22:57:03.760 INFO:tasks.rados.rados.0.smithi071.stdout:187: oid 3 updating version 28 to 29 2023-12-15T22:57:03.760 INFO:tasks.rados.rados.0.smithi071.stdout:187: oid 3 version 29 is already newer than 27 2023-12-15T22:57:03.760 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 29 (ObjNum 119 snap 10 seq_num 119) dirty exists 2023-12-15T22:57:03.760 INFO:tasks.rados.rados.0.smithi071.stdout:187: left oid 3 (ObjNum 119 snap 10 seq_num 119) 2023-12-15T22:57:03.760 INFO:tasks.rados.rados.0.smithi071.stdout:185: finishing write tid 4 to smithi07183189-4 2023-12-15T22:57:03.760 INFO:tasks.rados.rados.0.smithi071.stdout:188: finishing write tid 3 to smithi07183189-26 2023-12-15T22:57:03.762 INFO:tasks.rados.rados.0.smithi071.stdout:188: finishing write tid 4 to smithi07183189-26 2023-12-15T22:57:03.762 INFO:tasks.rados.rados.0.smithi071.stdout:188: oid 26 updating version 0 to 38 2023-12-15T22:57:03.762 INFO:tasks.rados.rados.0.smithi071.stdout:188: oid 26 updating version 38 to 40 2023-12-15T22:57:03.762 INFO:tasks.rados.rados.0.smithi071.stdout:188: oid 26 version 40 is already newer than 39 2023-12-15T22:57:03.762 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 40 (ObjNum 120 snap 10 seq_num 120) dirty exists 2023-12-15T22:57:03.762 INFO:tasks.rados.rados.0.smithi071.stdout:188: left oid 26 (ObjNum 120 snap 10 seq_num 120) 2023-12-15T22:57:03.762 INFO:tasks.rados.rados.0.smithi071.stdout:185: finishing write tid 5 to smithi07183189-4 2023-12-15T22:57:03.762 INFO:tasks.rados.rados.0.smithi071.stdout:185: oid 4 updating version 0 to 56 2023-12-15T22:57:03.763 INFO:tasks.rados.rados.0.smithi071.stdout:185: oid 4 version 56 is already newer than 54 2023-12-15T22:57:03.763 INFO:tasks.rados.rados.0.smithi071.stdout:185: oid 4 version 56 is already newer than 55 2023-12-15T22:57:03.763 INFO:tasks.rados.rados.0.smithi071.stdout:185: oid 4 updating version 56 to 57 2023-12-15T22:57:03.763 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 4 v 57 (ObjNum 117 snap 10 seq_num 117) dirty exists 2023-12-15T22:57:03.763 INFO:tasks.rados.rados.0.smithi071.stdout:185: left oid 4 (ObjNum 117 snap 10 seq_num 117) 2023-12-15T22:57:04.145 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:57:04.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:04.249+0000 7f44ad26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:04.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:04.254+0000 7f44ad26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:04.476 INFO:tasks.rados.rados.0.smithi071.stdout:184: done (4 left) 2023-12-15T22:57:04.476 INFO:tasks.rados.rados.0.smithi071.stdout:185: done (3 left) 2023-12-15T22:57:04.477 INFO:tasks.rados.rados.0.smithi071.stdout:187: done (2 left) 2023-12-15T22:57:04.477 INFO:tasks.rados.rados.0.smithi071.stdout:188: done (1 left) 2023-12-15T22:57:04.477 INFO:tasks.rados.rados.0.smithi071.stdout:189: done (0 left) 2023-12-15T22:57:04.477 INFO:tasks.rados.rados.0.smithi071.stdout:190: snap_create 2023-12-15T22:57:04.811 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T22:57:04.812 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2023-12-15T22:57:04.919 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:04.914+0000 7ff9b7413640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:04.922 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:04.922+0000 7ff9b7413640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:05.478 INFO:teuthology.orchestra.run.smithi071.stderr:reweighted osd.0 to 1 (10000) 2023-12-15T22:57:05.481 INFO:tasks.rados.rados.0.smithi071.stdout:190: done (0 left) 2023-12-15T22:57:05.481 INFO:tasks.rados.rados.0.smithi071.stdout:191: read oid 2 snap -1 2023-12-15T22:57:05.481 INFO:tasks.rados.rados.0.smithi071.stdout:191: expect (ObjNum 106 snap 9 seq_num 106) 2023-12-15T22:57:05.481 INFO:tasks.rados.rados.0.smithi071.stdout:192: snap_create 2023-12-15T22:57:05.495 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2023-12-15T22:57:05.530 INFO:tasks.rados.rados.0.smithi071.stdout:191: expect (ObjNum 106 snap 9 seq_num 106) 2023-12-15T22:57:05.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:05.599+0000 7f6ec1824640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:05.608 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:05.608+0000 7f6ec1824640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:06.322 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:57:06.427 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:06.426+0000 7fd1e475e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:06.430 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:06.430+0000 7fd1e475e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:06.479 INFO:teuthology.orchestra.run.smithi071.stderr:reweighted osd.1 to 1 (10000) 2023-12-15T22:57:06.482 INFO:tasks.rados.rados.0.smithi071.stdout:191: done (1 left) 2023-12-15T22:57:06.482 INFO:tasks.rados.rados.0.smithi071.stdout:192: done (0 left) 2023-12-15T22:57:06.482 INFO:tasks.rados.rados.0.smithi071.stdout:193: snap_create 2023-12-15T22:57:06.494 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2023-12-15T22:57:06.601 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:06.596+0000 7fe378e13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:06.604 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:06.604+0000 7fe378e13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:07.479 INFO:teuthology.orchestra.run.smithi071.stderr:reweighted osd.2 to 1 (10000) 2023-12-15T22:57:07.482 INFO:tasks.rados.rados.0.smithi071.stdout:193: done (0 left) 2023-12-15T22:57:07.482 INFO:tasks.rados.rados.0.smithi071.stdout:194: delete oid 46 current snap is 14 2023-12-15T22:57:07.493 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2023-12-15T22:57:07.524 INFO:tasks.rados.rados.0.smithi071.stdout:194: done (0 left) 2023-12-15T22:57:07.524 INFO:tasks.rados.rados.0.smithi071.stdout:195: snap_remove snap 13 2023-12-15T22:57:07.604 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:07.598+0000 7fc62011e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:07.607 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:07.607+0000 7fc62011e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:08.492 INFO:teuthology.orchestra.run.smithi071.stderr:reweighted osd.3 to 1 (10000) 2023-12-15T22:57:08.497 INFO:tasks.rados.rados.0.smithi071.stdout:195: done (0 left) 2023-12-15T22:57:08.497 INFO:tasks.rados.rados.0.smithi071.stdout:196: write (excl) oid 40 current snap is 14 2023-12-15T22:57:08.497 INFO:tasks.rados.rados.0.smithi071.stdout:196: seq_num 122 ranges {450454=642026,1505862=544592} 2023-12-15T22:57:08.502 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:57:08.504 INFO:tasks.rados.rados.0.smithi071.stdout:196: writing smithi07183189-40 from 450454 to 1092480 tid 1 2023-12-15T22:57:08.507 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2023-12-15T22:57:08.507 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T22:57:08.509 INFO:tasks.rados.rados.0.smithi071.stdout:196: writing smithi07183189-40 from 1505862 to 2050454 tid 2 2023-12-15T22:57:08.509 INFO:tasks.rados.rados.0.smithi071.stdout:197: read oid 8 snap 12 2023-12-15T22:57:08.510 INFO:tasks.rados.rados.0.smithi071.stdout:197: expect (ObjNum 7 snap 0 seq_num 7) 2023-12-15T22:57:08.510 INFO:tasks.rados.rados.0.smithi071.stdout:198: delete oid 17 current snap is 14 2023-12-15T22:57:08.578 INFO:tasks.rados.rados.0.smithi071.stdout:197: expect (ObjNum 7 snap 0 seq_num 7) 2023-12-15T22:57:08.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:08.630+0000 7f19bcd9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:08.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:08.637+0000 7f19bcd9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:08.641 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:08.639+0000 7f17c902d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:08.647 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:08.647+0000 7f17c882c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:08.704 INFO:tasks.rados.rados.0.smithi071.stdout:196: finishing write tid 1 to smithi07183189-40 2023-12-15T22:57:08.704 INFO:tasks.rados.rados.0.smithi071.stdout:196: finishing write tid 2 to smithi07183189-40 2023-12-15T22:57:08.704 INFO:tasks.rados.rados.0.smithi071.stdout:196: finishing write tid 3 to smithi07183189-40 2023-12-15T22:57:08.704 INFO:tasks.rados.rados.0.smithi071.stdout:196: finishing write tid 4 to smithi07183189-40 2023-12-15T22:57:08.704 INFO:tasks.rados.rados.0.smithi071.stdout:196: oid 40 updating version 0 to 82 2023-12-15T22:57:08.704 INFO:tasks.rados.rados.0.smithi071.stdout:196: oid 40 version 82 is already newer than 80 2023-12-15T22:57:08.704 INFO:tasks.rados.rados.0.smithi071.stdout:196: oid 40 version 82 is already newer than 82 2023-12-15T22:57:08.704 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 40 v 82 (ObjNum 122 snap 14 seq_num 122) dirty exists 2023-12-15T22:57:08.704 INFO:tasks.rados.rados.0.smithi071.stdout:196: left oid 40 (ObjNum 122 snap 14 seq_num 122) 2023-12-15T22:57:08.704 INFO:tasks.rados.rados.0.smithi071.stdout:196: done (2 left) 2023-12-15T22:57:08.705 INFO:tasks.rados.rados.0.smithi071.stdout:197: done (1 left) 2023-12-15T22:57:08.705 INFO:tasks.rados.rados.0.smithi071.stdout:198: done (0 left) 2023-12-15T22:57:08.705 INFO:tasks.rados.rados.0.smithi071.stdout:199: snap_create 2023-12-15T22:57:08.828 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:57:08.829 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-12-15T22:57:08.840 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":55,"stamp":"2023-12-15T22:57:08.518505+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120086950,"num_objects":69,"num_object_clones":23,"num_object_copies":207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":545,"num_read_kb":255196,"num_write":583,"num_write_kb":131273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":557,"ondisk_log_size":557,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":321052,"kb_used_data":196068,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377166308,"statfs":{"total":386547056640,"available":386218299392,"internally_reserved":0,"allocated":200773632,"data_stored":200497461,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8403269,"num_objects":-1,"num_object_clones":3,"num_object_copies":-3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":52,"num_read_kb":44044,"num_write":59,"num_write_kb":18426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001062"},"pg_stats":[{"pgid":"3.f","version":"17'6","reported_seq":72,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-15T22:57:05.510026+0000","last_change":"2023-12-15T22:57:03.565255+0000","last_active":"2023-12-15T22:57:05.510026+0000","last_peered":"2023-12-15T22:57:05.510026+0000","last_clean":"2023-12-15T22:57:05.510026+0000","last_became_active":"2023-12-15T22:56:39.503797+0000","last_became_peered":"2023-12-15T22:56:39.503797+0000","last_unstale":"2023-12-15T22:57:05.510026+0000","last_undegraded":"2023-12-15T22:57:05.510026+0000","last_fullsized":"2023-12-15T22:57:05.510026+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4537824,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":4966,"num_write":8,"num_write_kb":2181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":57,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-15T22:57:03.541053+0000","last_change":"2023-12-15T22:57:03.551310+0000","last_active":"2023-12-15T22:57:03.541053+0000","last_peered":"2023-12-15T22:57:03.541053+0000","last_clean":"2023-12-15T22:57:03.541053+0000","last_became_active":"2023-12-15T22:56:39.552856+0000","last_became_peered":"2023-12-15T22:56:39.552856+0000","last_unstale":"2023-12-15T22:57:03.541053+0000","last_undegraded":"2023-12-15T22:57:03.541053+0000","last_fullsized":"2023-12-15T22:57:03.541053+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"37'57","reported_seq":138,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-15T22:57:03.763439+0000","last_change":"2023-12-15T22:57:03.763439+0000","last_active":"2023-12-15T22:57:03.763439+0000","last_peered":"2023-12-15T22:57:03.763439+0000","last_clean":"2023-12-15T22:57:03.763439+0000","last_became_active":"2023-12-15T22:56:39.495490+0000","last_became_peered":"2023-12-15T22:56:39.495490+0000","last_unstale":"2023-12-15T22:57:03.763439+0000","last_undegraded":"2023-12-15T22:57:03.763439+0000","last_fullsized":"2023-12-15T22:57:03.763439+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17193396,"num_objects":8,"num_object_clones":2,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":66,"num_read_kb":39970,"num_write":62,"num_write_kb":17507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"35'24","reported_seq":110,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-15T22:57:05.509319+0000","last_change":"2023-12-15T22:57:03.580863+0000","last_active":"2023-12-15T22:57:05.509319+0000","last_peered":"2023-12-15T22:57:05.509319+0000","last_clean":"2023-12-15T22:57:05.509319+0000","last_became_active":"2023-12-15T22:56:39.579234+0000","last_became_peered":"2023-12-15T22:56:39.579234+0000","last_unstale":"2023-12-15T22:57:05.509319+0000","last_undegraded":"2023-12-15T22:57:05.509319+0000","last_fullsized":"2023-12-15T22:57:05.509319+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9680733,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":48,"num_read_kb":37627,"num_write":31,"num_write_kb":8622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"35'20","reported_seq":81,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-15T22:57:04.484098+0000","last_change":"2023-12-15T22:57:03.490807+0000","last_active":"2023-12-15T22:57:04.484098+0000","last_peered":"2023-12-15T22:57:04.484098+0000","last_clean":"2023-12-15T22:57:04.484098+0000","last_became_active":"2023-12-15T22:56:39.572176+0000","last_became_peered":"2023-12-15T22:56:39.572176+0000","last_unstale":"2023-12-15T22:57:04.484098+0000","last_undegraded":"2023-12-15T22:57:04.484098+0000","last_fullsized":"2023-12-15T22:57:04.484098+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3310946,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":3685,"num_write":17,"num_write_kb":5025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"35'18","reported_seq":90,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-15T22:57:05.525262+0000","last_change":"2023-12-15T22:57:03.557767+0000","last_active":"2023-12-15T22:57:05.525262+0000","last_peered":"2023-12-15T22:57:05.525262+0000","last_clean":"2023-12-15T22:57:05.525262+0000","last_became_active":"2023-12-15T22:56:39.516361+0000","last_became_peered":"2023-12-15T22:56:39.516361+0000","last_unstale":"2023-12-15T22:57:05.525262+0000","last_undegraded":"2023-12-15T22:57:05.525262+0000","last_fullsized":"2023-12-15T22:57:05.525262+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6067261,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":19605,"num_write":23,"num_write_kb":6799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"37'79","reported_seq":155,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-15T22:57:05.500012+0000","last_change":"2023-12-15T22:57:03.708613+0000","last_active":"2023-12-15T22:57:05.500012+0000","last_peered":"2023-12-15T22:57:05.500012+0000","last_clean":"2023-12-15T22:57:05.500012+0000","last_became_active":"2023-12-15T22:56:39.519748+0000","last_became_peered":"2023-12-15T22:56:39.519748+0000","last_unstale":"2023-12-15T22:57:05.500012+0000","last_undegraded":"2023-12-15T22:57:05.500012+0000","last_fullsized":"2023-12-15T22:57:05.500012+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16201783,"num_objects":13,"num_object_clones":6,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":75,"num_read_kb":42695,"num_write":66,"num_write_kb":19684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"37'29","reported_seq":87,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-15T22:57:03.729212+0000","last_change":"2023-12-15T22:57:03.538716+0000","last_active":"2023-12-15T22:57:03.729212+0000","last_peered":"2023-12-15T22:57:03.729212+0000","last_clean":"2023-12-15T22:57:03.729212+0000","last_became_active":"2023-12-15T22:56:39.543179+0000","last_became_peered":"2023-12-15T22:56:39.543179+0000","last_unstale":"2023-12-15T22:57:03.729212+0000","last_undegraded":"2023-12-15T22:57:03.729212+0000","last_fullsized":"2023-12-15T22:57:03.729212+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8016873,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":8864,"num_write":31,"num_write_kb":7951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":182,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-15T22:57:04.490196+0000","last_change":"2023-12-15T22:56:32.607017+0000","last_active":"2023-12-15T22:57:04.490196+0000","last_peered":"2023-12-15T22:57:04.490196+0000","last_clean":"2023-12-15T22:57:04.490196+0000","last_became_active":"2023-12-15T22:56:32.602384+0000","last_became_peered":"2023-12-15T22:56:32.602384+0000","last_unstale":"2023-12-15T22:57:04.490196+0000","last_undegraded":"2023-12-15T22:57:04.490196+0000","last_fullsized":"2023-12-15T22:57:04.490196+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_clean_scrub_stamp":"2023-12-15T22:56:31.525572+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":51,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-15T22:57:04.550538+0000","last_change":"2023-12-15T22:56:33.627481+0000","last_active":"2023-12-15T22:57:04.550538+0000","last_peered":"2023-12-15T22:57:04.550538+0000","last_clean":"2023-12-15T22:57:04.550538+0000","last_became_active":"2023-12-15T22:56:33.624689+0000","last_became_peered":"2023-12-15T22:56:33.624689+0000","last_unstale":"2023-12-15T22:57:04.550538+0000","last_undegraded":"2023-12-15T22:57:04.550538+0000","last_fullsized":"2023-12-15T22:57:04.550538+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"35'38","reported_seq":104,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-15T22:57:03.550909+0000","last_change":"2023-12-15T22:57:03.550909+0000","last_active":"2023-12-15T22:57:03.550909+0000","last_peered":"2023-12-15T22:57:03.550909+0000","last_clean":"2023-12-15T22:57:03.550909+0000","last_became_active":"2023-12-15T22:56:39.529359+0000","last_became_peered":"2023-12-15T22:56:39.529359+0000","last_unstale":"2023-12-15T22:57:03.550909+0000","last_undegraded":"2023-12-15T22:57:03.550909+0000","last_fullsized":"2023-12-15T22:57:03.550909+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7294172,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":30,"num_read_kb":10700,"num_write":40,"num_write_kb":12175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":57,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-15T22:57:05.499293+0000","last_change":"2023-12-15T22:56:33.655797+0000","last_active":"2023-12-15T22:57:05.499293+0000","last_peered":"2023-12-15T22:57:05.499293+0000","last_clean":"2023-12-15T22:57:05.499293+0000","last_became_active":"2023-12-15T22:56:33.652885+0000","last_became_peered":"2023-12-15T22:56:33.652885+0000","last_unstale":"2023-12-15T22:57:05.499293+0000","last_undegraded":"2023-12-15T22:57:05.499293+0000","last_fullsized":"2023-12-15T22:57:05.499293+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"37'37","reported_seq":108,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-15T22:57:04.492616+0000","last_change":"2023-12-15T22:57:03.532864+0000","last_active":"2023-12-15T22:57:04.492616+0000","last_peered":"2023-12-15T22:57:04.492616+0000","last_clean":"2023-12-15T22:57:04.492616+0000","last_became_active":"2023-12-15T22:56:39.543509+0000","last_became_peered":"2023-12-15T22:56:39.543509+0000","last_unstale":"2023-12-15T22:57:04.492616+0000","last_undegraded":"2023-12-15T22:57:04.492616+0000","last_fullsized":"2023-12-15T22:57:04.492616+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12504276,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":39,"num_read_kb":21611,"num_write":35,"num_write_kb":9506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":51,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-15T22:57:04.549603+0000","last_change":"2023-12-15T22:56:33.600485+0000","last_active":"2023-12-15T22:57:04.549603+0000","last_peered":"2023-12-15T22:57:04.549603+0000","last_clean":"2023-12-15T22:57:04.549603+0000","last_became_active":"2023-12-15T22:56:33.597320+0000","last_became_peered":"2023-12-15T22:56:33.597320+0000","last_unstale":"2023-12-15T22:57:04.549603+0000","last_undegraded":"2023-12-15T22:57:04.549603+0000","last_fullsized":"2023-12-15T22:57:04.549603+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"37'23","reported_seq":83,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-15T22:57:04.486176+0000","last_change":"2023-12-15T22:57:03.494489+0000","last_active":"2023-12-15T22:57:04.486176+0000","last_peered":"2023-12-15T22:57:04.486176+0000","last_clean":"2023-12-15T22:57:04.486176+0000","last_became_active":"2023-12-15T22:56:39.560980+0000","last_became_peered":"2023-12-15T22:56:39.560980+0000","last_unstale":"2023-12-15T22:57:04.486176+0000","last_undegraded":"2023-12-15T22:57:04.486176+0000","last_fullsized":"2023-12-15T22:57:04.486176+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3226911,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":13739,"num_write":17,"num_write_kb":4752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":57,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-15T22:57:05.508390+0000","last_change":"2023-12-15T22:56:33.648541+0000","last_active":"2023-12-15T22:57:05.508390+0000","last_peered":"2023-12-15T22:57:05.508390+0000","last_clean":"2023-12-15T22:57:05.508390+0000","last_became_active":"2023-12-15T22:56:33.643681+0000","last_became_peered":"2023-12-15T22:56:33.643681+0000","last_unstale":"2023-12-15T22:57:05.508390+0000","last_undegraded":"2023-12-15T22:57:05.508390+0000","last_fullsized":"2023-12-15T22:57:05.508390+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"35'43","reported_seq":110,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-15T22:57:03.545412+0000","last_change":"2023-12-15T22:57:03.555730+0000","last_active":"2023-12-15T22:57:03.545412+0000","last_peered":"2023-12-15T22:57:03.545412+0000","last_clean":"2023-12-15T22:57:03.545412+0000","last_became_active":"2023-12-15T22:56:39.506510+0000","last_became_peered":"2023-12-15T22:56:39.506510+0000","last_unstale":"2023-12-15T22:57:03.545412+0000","last_undegraded":"2023-12-15T22:57:03.545412+0000","last_fullsized":"2023-12-15T22:57:03.545412+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13563117,"num_objects":8,"num_object_clones":3,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":39,"num_read_kb":20283,"num_write":40,"num_write_kb":12077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"17'7","reported_seq":64,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-15T22:57:04.550361+0000","last_change":"2023-12-15T22:57:03.569354+0000","last_active":"2023-12-15T22:57:04.550361+0000","last_peered":"2023-12-15T22:57:04.550361+0000","last_clean":"2023-12-15T22:57:04.550361+0000","last_became_active":"2023-12-15T22:56:39.511159+0000","last_became_peered":"2023-12-15T22:56:39.511159+0000","last_unstale":"2023-12-15T22:57:04.550361+0000","last_undegraded":"2023-12-15T22:57:04.550361+0000","last_fullsized":"2023-12-15T22:57:04.550361+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2941192,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":9,"num_write_kb":2133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":53,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-15T22:57:04.545851+0000","last_change":"2023-12-15T22:56:33.633890+0000","last_active":"2023-12-15T22:57:04.545851+0000","last_peered":"2023-12-15T22:57:04.545851+0000","last_clean":"2023-12-15T22:57:04.545851+0000","last_became_active":"2023-12-15T22:56:33.631134+0000","last_became_peered":"2023-12-15T22:56:33.631134+0000","last_unstale":"2023-12-15T22:57:04.545851+0000","last_undegraded":"2023-12-15T22:57:04.545851+0000","last_fullsized":"2023-12-15T22:57:04.545851+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"35'36","reported_seq":89,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-15T22:57:04.544860+0000","last_change":"2023-12-15T22:57:03.555425+0000","last_active":"2023-12-15T22:57:04.544860+0000","last_peered":"2023-12-15T22:57:04.544860+0000","last_clean":"2023-12-15T22:57:04.544860+0000","last_became_active":"2023-12-15T22:56:39.532730+0000","last_became_peered":"2023-12-15T22:56:39.532730+0000","last_unstale":"2023-12-15T22:57:04.544860+0000","last_undegraded":"2023-12-15T22:57:04.544860+0000","last_fullsized":"2023-12-15T22:57:04.544860+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6806769,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":5025,"num_write":27,"num_write_kb":8830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":53,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-15T22:57:04.491929+0000","last_change":"2023-12-15T22:56:33.655736+0000","last_active":"2023-12-15T22:57:04.491929+0000","last_peered":"2023-12-15T22:57:04.491929+0000","last_clean":"2023-12-15T22:57:04.491929+0000","last_became_active":"2023-12-15T22:56:33.652917+0000","last_became_peered":"2023-12-15T22:56:33.652917+0000","last_unstale":"2023-12-15T22:57:04.491929+0000","last_undegraded":"2023-12-15T22:57:04.491929+0000","last_fullsized":"2023-12-15T22:57:04.491929+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"37'40","reported_seq":107,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-15T22:57:04.488662+0000","last_change":"2023-12-15T22:57:03.495085+0000","last_active":"2023-12-15T22:57:04.488662+0000","last_peered":"2023-12-15T22:57:04.488662+0000","last_clean":"2023-12-15T22:57:04.488662+0000","last_became_active":"2023-12-15T22:56:39.557252+0000","last_became_peered":"2023-12-15T22:56:39.557252+0000","last_unstale":"2023-12-15T22:57:04.488662+0000","last_undegraded":"2023-12-15T22:57:04.488662+0000","last_fullsized":"2023-12-15T22:57:04.488662+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5596698,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":33,"num_read_kb":18847,"num_write":36,"num_write_kb":11126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":53,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-15T22:57:04.544416+0000","last_change":"2023-12-15T22:56:33.632348+0000","last_active":"2023-12-15T22:57:04.544416+0000","last_peered":"2023-12-15T22:57:04.544416+0000","last_clean":"2023-12-15T22:57:04.544416+0000","last_became_active":"2023-12-15T22:56:33.629022+0000","last_became_peered":"2023-12-15T22:56:33.629022+0000","last_unstale":"2023-12-15T22:57:04.544416+0000","last_undegraded":"2023-12-15T22:57:04.544416+0000","last_fullsized":"2023-12-15T22:57:04.544416+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"17'3","reported_seq":64,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-15T22:57:03.544304+0000","last_change":"2023-12-15T22:57:03.551777+0000","last_active":"2023-12-15T22:57:03.544304+0000","last_peered":"2023-12-15T22:57:03.544304+0000","last_clean":"2023-12-15T22:57:03.544304+0000","last_became_active":"2023-12-15T22:56:39.535145+0000","last_became_peered":"2023-12-15T22:56:39.535145+0000","last_unstale":"2023-12-15T22:57:03.544304+0000","last_undegraded":"2023-12-15T22:57:03.544304+0000","last_fullsized":"2023-12-15T22:57:03.544304+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2554631,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":7486,"num_write":4,"num_write_kb":1011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":51,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-15T22:57:04.548365+0000","last_change":"2023-12-15T22:56:33.627005+0000","last_active":"2023-12-15T22:57:04.548365+0000","last_peered":"2023-12-15T22:57:04.548365+0000","last_clean":"2023-12-15T22:57:04.548365+0000","last_became_active":"2023-12-15T22:56:33.622181+0000","last_became_peered":"2023-12-15T22:56:33.622181+0000","last_unstale":"2023-12-15T22:57:04.548365+0000","last_undegraded":"2023-12-15T22:57:04.548365+0000","last_fullsized":"2023-12-15T22:57:04.548365+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":119496582,"num_objects":67,"num_object_clones":23,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":442,"num_read_kb":255105,"num_write":446,"num_write_kb":129379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":460,"ondisk_log_size":460,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705700,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":83932,"kb_used_data":52232,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94287908,"statfs":{"total":96636764160,"available":96550817792,"internally_reserved":0,"allocated":53485568,"data_stored":53503457,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705698,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":71272,"kb_used_data":40120,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94300568,"statfs":{"total":96636764160,"available":96563781632,"internally_reserved":0,"allocated":41082880,"data_stored":41179881,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705698,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":78876,"kb_used_data":47160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94292964,"statfs":{"total":96636764160,"available":96555995136,"internally_reserved":0,"allocated":48291840,"data_stored":47902192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705699,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":86972,"kb_used_data":56556,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94284868,"statfs":{"total":96636764160,"available":96547704832,"internally_reserved":0,"allocated":57913344,"data_stored":57911931,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-15T22:57:08.842 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T22:57:08.946 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:08.943+0000 7f719f22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:08.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:08.949+0000 7f719f22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:09.123 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:57:09.123 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-12-15T22:57:09.134 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":55,"stamp":"2023-12-15T22:57:08.518505+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120086950,"num_objects":69,"num_object_clones":23,"num_object_copies":207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":545,"num_read_kb":255196,"num_write":583,"num_write_kb":131273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":557,"ondisk_log_size":557,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":321052,"kb_used_data":196068,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377166308,"statfs":{"total":386547056640,"available":386218299392,"internally_reserved":0,"allocated":200773632,"data_stored":200497461,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8403269,"num_objects":-1,"num_object_clones":3,"num_object_copies":-3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":52,"num_read_kb":44044,"num_write":59,"num_write_kb":18426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001062"},"pg_stats":[{"pgid":"3.f","version":"17'6","reported_seq":72,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-15T22:57:05.510026+0000","last_change":"2023-12-15T22:57:03.565255+0000","last_active":"2023-12-15T22:57:05.510026+0000","last_peered":"2023-12-15T22:57:05.510026+0000","last_clean":"2023-12-15T22:57:05.510026+0000","last_became_active":"2023-12-15T22:56:39.503797+0000","last_became_peered":"2023-12-15T22:56:39.503797+0000","last_unstale":"2023-12-15T22:57:05.510026+0000","last_undegraded":"2023-12-15T22:57:05.510026+0000","last_fullsized":"2023-12-15T22:57:05.510026+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4537824,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":4966,"num_write":8,"num_write_kb":2181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":57,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-15T22:57:03.541053+0000","last_change":"2023-12-15T22:57:03.551310+0000","last_active":"2023-12-15T22:57:03.541053+0000","last_peered":"2023-12-15T22:57:03.541053+0000","last_clean":"2023-12-15T22:57:03.541053+0000","last_became_active":"2023-12-15T22:56:39.552856+0000","last_became_peered":"2023-12-15T22:56:39.552856+0000","last_unstale":"2023-12-15T22:57:03.541053+0000","last_undegraded":"2023-12-15T22:57:03.541053+0000","last_fullsized":"2023-12-15T22:57:03.541053+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"37'57","reported_seq":138,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-15T22:57:03.763439+0000","last_change":"2023-12-15T22:57:03.763439+0000","last_active":"2023-12-15T22:57:03.763439+0000","last_peered":"2023-12-15T22:57:03.763439+0000","last_clean":"2023-12-15T22:57:03.763439+0000","last_became_active":"2023-12-15T22:56:39.495490+0000","last_became_peered":"2023-12-15T22:56:39.495490+0000","last_unstale":"2023-12-15T22:57:03.763439+0000","last_undegraded":"2023-12-15T22:57:03.763439+0000","last_fullsized":"2023-12-15T22:57:03.763439+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17193396,"num_objects":8,"num_object_clones":2,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":66,"num_read_kb":39970,"num_write":62,"num_write_kb":17507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"35'24","reported_seq":110,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-15T22:57:05.509319+0000","last_change":"2023-12-15T22:57:03.580863+0000","last_active":"2023-12-15T22:57:05.509319+0000","last_peered":"2023-12-15T22:57:05.509319+0000","last_clean":"2023-12-15T22:57:05.509319+0000","last_became_active":"2023-12-15T22:56:39.579234+0000","last_became_peered":"2023-12-15T22:56:39.579234+0000","last_unstale":"2023-12-15T22:57:05.509319+0000","last_undegraded":"2023-12-15T22:57:05.509319+0000","last_fullsized":"2023-12-15T22:57:05.509319+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9680733,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":48,"num_read_kb":37627,"num_write":31,"num_write_kb":8622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"35'20","reported_seq":81,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-15T22:57:04.484098+0000","last_change":"2023-12-15T22:57:03.490807+0000","last_active":"2023-12-15T22:57:04.484098+0000","last_peered":"2023-12-15T22:57:04.484098+0000","last_clean":"2023-12-15T22:57:04.484098+0000","last_became_active":"2023-12-15T22:56:39.572176+0000","last_became_peered":"2023-12-15T22:56:39.572176+0000","last_unstale":"2023-12-15T22:57:04.484098+0000","last_undegraded":"2023-12-15T22:57:04.484098+0000","last_fullsized":"2023-12-15T22:57:04.484098+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3310946,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":3685,"num_write":17,"num_write_kb":5025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"35'18","reported_seq":90,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-15T22:57:05.525262+0000","last_change":"2023-12-15T22:57:03.557767+0000","last_active":"2023-12-15T22:57:05.525262+0000","last_peered":"2023-12-15T22:57:05.525262+0000","last_clean":"2023-12-15T22:57:05.525262+0000","last_became_active":"2023-12-15T22:56:39.516361+0000","last_became_peered":"2023-12-15T22:56:39.516361+0000","last_unstale":"2023-12-15T22:57:05.525262+0000","last_undegraded":"2023-12-15T22:57:05.525262+0000","last_fullsized":"2023-12-15T22:57:05.525262+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6067261,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":19605,"num_write":23,"num_write_kb":6799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"37'79","reported_seq":155,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-15T22:57:05.500012+0000","last_change":"2023-12-15T22:57:03.708613+0000","last_active":"2023-12-15T22:57:05.500012+0000","last_peered":"2023-12-15T22:57:05.500012+0000","last_clean":"2023-12-15T22:57:05.500012+0000","last_became_active":"2023-12-15T22:56:39.519748+0000","last_became_peered":"2023-12-15T22:56:39.519748+0000","last_unstale":"2023-12-15T22:57:05.500012+0000","last_undegraded":"2023-12-15T22:57:05.500012+0000","last_fullsized":"2023-12-15T22:57:05.500012+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16201783,"num_objects":13,"num_object_clones":6,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":75,"num_read_kb":42695,"num_write":66,"num_write_kb":19684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"37'29","reported_seq":87,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-15T22:57:03.729212+0000","last_change":"2023-12-15T22:57:03.538716+0000","last_active":"2023-12-15T22:57:03.729212+0000","last_peered":"2023-12-15T22:57:03.729212+0000","last_clean":"2023-12-15T22:57:03.729212+0000","last_became_active":"2023-12-15T22:56:39.543179+0000","last_became_peered":"2023-12-15T22:56:39.543179+0000","last_unstale":"2023-12-15T22:57:03.729212+0000","last_undegraded":"2023-12-15T22:57:03.729212+0000","last_fullsized":"2023-12-15T22:57:03.729212+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8016873,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":8864,"num_write":31,"num_write_kb":7951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":182,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-15T22:57:04.490196+0000","last_change":"2023-12-15T22:56:32.607017+0000","last_active":"2023-12-15T22:57:04.490196+0000","last_peered":"2023-12-15T22:57:04.490196+0000","last_clean":"2023-12-15T22:57:04.490196+0000","last_became_active":"2023-12-15T22:56:32.602384+0000","last_became_peered":"2023-12-15T22:56:32.602384+0000","last_unstale":"2023-12-15T22:57:04.490196+0000","last_undegraded":"2023-12-15T22:57:04.490196+0000","last_fullsized":"2023-12-15T22:57:04.490196+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_clean_scrub_stamp":"2023-12-15T22:56:31.525572+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":51,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-15T22:57:04.550538+0000","last_change":"2023-12-15T22:56:33.627481+0000","last_active":"2023-12-15T22:57:04.550538+0000","last_peered":"2023-12-15T22:57:04.550538+0000","last_clean":"2023-12-15T22:57:04.550538+0000","last_became_active":"2023-12-15T22:56:33.624689+0000","last_became_peered":"2023-12-15T22:56:33.624689+0000","last_unstale":"2023-12-15T22:57:04.550538+0000","last_undegraded":"2023-12-15T22:57:04.550538+0000","last_fullsized":"2023-12-15T22:57:04.550538+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"35'38","reported_seq":104,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-15T22:57:03.550909+0000","last_change":"2023-12-15T22:57:03.550909+0000","last_active":"2023-12-15T22:57:03.550909+0000","last_peered":"2023-12-15T22:57:03.550909+0000","last_clean":"2023-12-15T22:57:03.550909+0000","last_became_active":"2023-12-15T22:56:39.529359+0000","last_became_peered":"2023-12-15T22:56:39.529359+0000","last_unstale":"2023-12-15T22:57:03.550909+0000","last_undegraded":"2023-12-15T22:57:03.550909+0000","last_fullsized":"2023-12-15T22:57:03.550909+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7294172,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":30,"num_read_kb":10700,"num_write":40,"num_write_kb":12175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":57,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-15T22:57:05.499293+0000","last_change":"2023-12-15T22:56:33.655797+0000","last_active":"2023-12-15T22:57:05.499293+0000","last_peered":"2023-12-15T22:57:05.499293+0000","last_clean":"2023-12-15T22:57:05.499293+0000","last_became_active":"2023-12-15T22:56:33.652885+0000","last_became_peered":"2023-12-15T22:56:33.652885+0000","last_unstale":"2023-12-15T22:57:05.499293+0000","last_undegraded":"2023-12-15T22:57:05.499293+0000","last_fullsized":"2023-12-15T22:57:05.499293+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"37'37","reported_seq":108,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-15T22:57:04.492616+0000","last_change":"2023-12-15T22:57:03.532864+0000","last_active":"2023-12-15T22:57:04.492616+0000","last_peered":"2023-12-15T22:57:04.492616+0000","last_clean":"2023-12-15T22:57:04.492616+0000","last_became_active":"2023-12-15T22:56:39.543509+0000","last_became_peered":"2023-12-15T22:56:39.543509+0000","last_unstale":"2023-12-15T22:57:04.492616+0000","last_undegraded":"2023-12-15T22:57:04.492616+0000","last_fullsized":"2023-12-15T22:57:04.492616+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12504276,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":39,"num_read_kb":21611,"num_write":35,"num_write_kb":9506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":51,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-15T22:57:04.549603+0000","last_change":"2023-12-15T22:56:33.600485+0000","last_active":"2023-12-15T22:57:04.549603+0000","last_peered":"2023-12-15T22:57:04.549603+0000","last_clean":"2023-12-15T22:57:04.549603+0000","last_became_active":"2023-12-15T22:56:33.597320+0000","last_became_peered":"2023-12-15T22:56:33.597320+0000","last_unstale":"2023-12-15T22:57:04.549603+0000","last_undegraded":"2023-12-15T22:57:04.549603+0000","last_fullsized":"2023-12-15T22:57:04.549603+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"37'23","reported_seq":83,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-15T22:57:04.486176+0000","last_change":"2023-12-15T22:57:03.494489+0000","last_active":"2023-12-15T22:57:04.486176+0000","last_peered":"2023-12-15T22:57:04.486176+0000","last_clean":"2023-12-15T22:57:04.486176+0000","last_became_active":"2023-12-15T22:56:39.560980+0000","last_became_peered":"2023-12-15T22:56:39.560980+0000","last_unstale":"2023-12-15T22:57:04.486176+0000","last_undegraded":"2023-12-15T22:57:04.486176+0000","last_fullsized":"2023-12-15T22:57:04.486176+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3226911,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":13739,"num_write":17,"num_write_kb":4752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":57,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-15T22:57:05.508390+0000","last_change":"2023-12-15T22:56:33.648541+0000","last_active":"2023-12-15T22:57:05.508390+0000","last_peered":"2023-12-15T22:57:05.508390+0000","last_clean":"2023-12-15T22:57:05.508390+0000","last_became_active":"2023-12-15T22:56:33.643681+0000","last_became_peered":"2023-12-15T22:56:33.643681+0000","last_unstale":"2023-12-15T22:57:05.508390+0000","last_undegraded":"2023-12-15T22:57:05.508390+0000","last_fullsized":"2023-12-15T22:57:05.508390+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"35'43","reported_seq":110,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-15T22:57:03.545412+0000","last_change":"2023-12-15T22:57:03.555730+0000","last_active":"2023-12-15T22:57:03.545412+0000","last_peered":"2023-12-15T22:57:03.545412+0000","last_clean":"2023-12-15T22:57:03.545412+0000","last_became_active":"2023-12-15T22:56:39.506510+0000","last_became_peered":"2023-12-15T22:56:39.506510+0000","last_unstale":"2023-12-15T22:57:03.545412+0000","last_undegraded":"2023-12-15T22:57:03.545412+0000","last_fullsized":"2023-12-15T22:57:03.545412+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13563117,"num_objects":8,"num_object_clones":3,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":39,"num_read_kb":20283,"num_write":40,"num_write_kb":12077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"17'7","reported_seq":64,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-15T22:57:04.550361+0000","last_change":"2023-12-15T22:57:03.569354+0000","last_active":"2023-12-15T22:57:04.550361+0000","last_peered":"2023-12-15T22:57:04.550361+0000","last_clean":"2023-12-15T22:57:04.550361+0000","last_became_active":"2023-12-15T22:56:39.511159+0000","last_became_peered":"2023-12-15T22:56:39.511159+0000","last_unstale":"2023-12-15T22:57:04.550361+0000","last_undegraded":"2023-12-15T22:57:04.550361+0000","last_fullsized":"2023-12-15T22:57:04.550361+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2941192,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":9,"num_write_kb":2133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":53,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-15T22:57:04.545851+0000","last_change":"2023-12-15T22:56:33.633890+0000","last_active":"2023-12-15T22:57:04.545851+0000","last_peered":"2023-12-15T22:57:04.545851+0000","last_clean":"2023-12-15T22:57:04.545851+0000","last_became_active":"2023-12-15T22:56:33.631134+0000","last_became_peered":"2023-12-15T22:56:33.631134+0000","last_unstale":"2023-12-15T22:57:04.545851+0000","last_undegraded":"2023-12-15T22:57:04.545851+0000","last_fullsized":"2023-12-15T22:57:04.545851+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"35'36","reported_seq":89,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-15T22:57:04.544860+0000","last_change":"2023-12-15T22:57:03.555425+0000","last_active":"2023-12-15T22:57:04.544860+0000","last_peered":"2023-12-15T22:57:04.544860+0000","last_clean":"2023-12-15T22:57:04.544860+0000","last_became_active":"2023-12-15T22:56:39.532730+0000","last_became_peered":"2023-12-15T22:56:39.532730+0000","last_unstale":"2023-12-15T22:57:04.544860+0000","last_undegraded":"2023-12-15T22:57:04.544860+0000","last_fullsized":"2023-12-15T22:57:04.544860+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6806769,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":5025,"num_write":27,"num_write_kb":8830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":53,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-15T22:57:04.491929+0000","last_change":"2023-12-15T22:56:33.655736+0000","last_active":"2023-12-15T22:57:04.491929+0000","last_peered":"2023-12-15T22:57:04.491929+0000","last_clean":"2023-12-15T22:57:04.491929+0000","last_became_active":"2023-12-15T22:56:33.652917+0000","last_became_peered":"2023-12-15T22:56:33.652917+0000","last_unstale":"2023-12-15T22:57:04.491929+0000","last_undegraded":"2023-12-15T22:57:04.491929+0000","last_fullsized":"2023-12-15T22:57:04.491929+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"37'40","reported_seq":107,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-15T22:57:04.488662+0000","last_change":"2023-12-15T22:57:03.495085+0000","last_active":"2023-12-15T22:57:04.488662+0000","last_peered":"2023-12-15T22:57:04.488662+0000","last_clean":"2023-12-15T22:57:04.488662+0000","last_became_active":"2023-12-15T22:56:39.557252+0000","last_became_peered":"2023-12-15T22:56:39.557252+0000","last_unstale":"2023-12-15T22:57:04.488662+0000","last_undegraded":"2023-12-15T22:57:04.488662+0000","last_fullsized":"2023-12-15T22:57:04.488662+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5596698,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":33,"num_read_kb":18847,"num_write":36,"num_write_kb":11126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":53,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-15T22:57:04.544416+0000","last_change":"2023-12-15T22:56:33.632348+0000","last_active":"2023-12-15T22:57:04.544416+0000","last_peered":"2023-12-15T22:57:04.544416+0000","last_clean":"2023-12-15T22:57:04.544416+0000","last_became_active":"2023-12-15T22:56:33.629022+0000","last_became_peered":"2023-12-15T22:56:33.629022+0000","last_unstale":"2023-12-15T22:57:04.544416+0000","last_undegraded":"2023-12-15T22:57:04.544416+0000","last_fullsized":"2023-12-15T22:57:04.544416+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"17'3","reported_seq":64,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-15T22:57:03.544304+0000","last_change":"2023-12-15T22:57:03.551777+0000","last_active":"2023-12-15T22:57:03.544304+0000","last_peered":"2023-12-15T22:57:03.544304+0000","last_clean":"2023-12-15T22:57:03.544304+0000","last_became_active":"2023-12-15T22:56:39.535145+0000","last_became_peered":"2023-12-15T22:56:39.535145+0000","last_unstale":"2023-12-15T22:57:03.544304+0000","last_undegraded":"2023-12-15T22:57:03.544304+0000","last_fullsized":"2023-12-15T22:57:03.544304+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2554631,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":7486,"num_write":4,"num_write_kb":1011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":51,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-15T22:57:04.548365+0000","last_change":"2023-12-15T22:56:33.627005+0000","last_active":"2023-12-15T22:57:04.548365+0000","last_peered":"2023-12-15T22:57:04.548365+0000","last_clean":"2023-12-15T22:57:04.548365+0000","last_became_active":"2023-12-15T22:56:33.622181+0000","last_became_peered":"2023-12-15T22:56:33.622181+0000","last_unstale":"2023-12-15T22:57:04.548365+0000","last_undegraded":"2023-12-15T22:57:04.548365+0000","last_fullsized":"2023-12-15T22:57:04.548365+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":119496582,"num_objects":67,"num_object_clones":23,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":442,"num_read_kb":255105,"num_write":446,"num_write_kb":129379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":460,"ondisk_log_size":460,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705700,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":83932,"kb_used_data":52232,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94287908,"statfs":{"total":96636764160,"available":96550817792,"internally_reserved":0,"allocated":53485568,"data_stored":53503457,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705698,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":71272,"kb_used_data":40120,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94300568,"statfs":{"total":96636764160,"available":96563781632,"internally_reserved":0,"allocated":41082880,"data_stored":41179881,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705698,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":78876,"kb_used_data":47160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94292964,"statfs":{"total":96636764160,"available":96555995136,"internally_reserved":0,"allocated":48291840,"data_stored":47902192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705699,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":86972,"kb_used_data":56556,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94284868,"statfs":{"total":96636764160,"available":96547704832,"internally_reserved":0,"allocated":57913344,"data_stored":57911931,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-15T22:57:09.136 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2023-12-15T22:57:09.136 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T22:57:09.136 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-12-15T22:57:09.136 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-12-15T22:57:09.241 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:09.237+0000 7f5db571e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:09.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:09.244+0000 7f5db571e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:09.256 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:57:09.256 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:57:09.257 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:57:09.257 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:57:09.257 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:57:09.257 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:57:09.257 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:57:09.257 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:57:09.257 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:57:09.257 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:57:09.257 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:57:09.257 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:57:09.259 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:57:09.260 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:57:09.260 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-12-15T22:57:09.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:09.361+0000 7f8d4202d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:09.369 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:09.369+0000 7f8d4202d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:09.381 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:57:09.381 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:57:09.381 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:57:09.381 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:57:09.381 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:57:09.381 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:57:09.381 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:57:09.382 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:57:09.382 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:57:09.382 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:57:09.382 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:57:09.382 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:57:09.384 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:57:09.385 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:57:09.385 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-12-15T22:57:09.491 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:09.486+0000 7f56d259e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:09.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:09.496+0000 7f56d259e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:09.498 INFO:tasks.rados.rados.0.smithi071.stdout:199: done (0 left) 2023-12-15T22:57:09.498 INFO:tasks.rados.rados.0.smithi071.stdout:200: write (excl) oid 33 current snap is 15 2023-12-15T22:57:09.498 INFO:tasks.rados.rados.0.smithi071.stdout:200: seq_num 124 ranges {484068=792524,1906473=177595} 2023-12-15T22:57:09.506 INFO:tasks.rados.rados.0.smithi071.stdout:200: writing smithi07183189-33 from 484068 to 1276592 tid 1 2023-12-15T22:57:09.511 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:57:09.511 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:57:09.511 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:57:09.511 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:57:09.512 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:57:09.512 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:57:09.512 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:57:09.512 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:57:09.512 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:57:09.512 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:57:09.513 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:57:09.513 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:57:09.514 INFO:tasks.rados.rados.0.smithi071.stdout:200: writing smithi07183189-33 from 1906473 to 2084068 tid 2 2023-12-15T22:57:09.514 INFO:tasks.rados.rados.0.smithi071.stdout:201: read oid 46 snap 12 2023-12-15T22:57:09.514 INFO:tasks.rados.rados.0.smithi071.stdout:201: expect (ObjNum 62 snap 2 seq_num 62) 2023-12-15T22:57:09.514 INFO:tasks.rados.rados.0.smithi071.stdout:202: delete oid 19 current snap is 15 2023-12-15T22:57:09.515 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:57:09.516 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:57:09.516 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-12-15T22:57:09.564 INFO:tasks.rados.rados.0.smithi071.stdout:202: done (2 left) 2023-12-15T22:57:09.564 INFO:tasks.rados.rados.0.smithi071.stdout:203: snap_remove snap 14 2023-12-15T22:57:09.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:09.629+0000 7f72b542d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:09.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:09.636+0000 7f72b542d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:09.649 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:57:09.649 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:57:09.649 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:57:09.650 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:57:09.650 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:57:09.650 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:57:09.650 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:57:09.650 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:57:09.650 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:57:09.650 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:57:09.650 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:57:09.650 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:57:09.653 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:57:09.653 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:57:10.503 INFO:tasks.rados.rados.0.smithi071.stdout:203: done (2 left) 2023-12-15T22:57:10.503 INFO:tasks.rados.rados.0.smithi071.stdout:204: read oid 32 snap -1 2023-12-15T22:57:10.503 INFO:tasks.rados.rados.0.smithi071.stdout:204: expect (ObjNum 31 snap 0 seq_num 31) 2023-12-15T22:57:10.503 INFO:tasks.rados.rados.0.smithi071.stdout:205: write (excl) oid 5 current snap is 15 2023-12-15T22:57:10.503 INFO:tasks.rados.rados.0.smithi071.stdout:205: seq_num 126 ranges {517682=543022,1623437=494245} 2023-12-15T22:57:10.509 INFO:tasks.rados.rados.0.smithi071.stdout:205: writing smithi07183189-5 from 517682 to 1060704 tid 1 2023-12-15T22:57:10.515 INFO:tasks.rados.rados.0.smithi071.stdout:205: writing smithi07183189-5 from 1623437 to 2117682 tid 2 2023-12-15T22:57:10.515 INFO:tasks.rados.rados.0.smithi071.stdout:206: read oid 31 snap -1 2023-12-15T22:57:10.515 INFO:tasks.rados.rados.0.smithi071.stdout:206: expect (ObjNum 94 snap 7 seq_num 94) 2023-12-15T22:57:10.515 INFO:tasks.rados.rados.0.smithi071.stdout:207: snap_remove snap 10 2023-12-15T22:57:10.743 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:57:10.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:10.846+0000 7f6e0bede640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:10.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:10.850+0000 7f6e0bede640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:11.507 INFO:tasks.rados.rados.0.smithi071.stdout:207: done (5 left) 2023-12-15T22:57:11.507 INFO:tasks.rados.rados.0.smithi071.stdout:208: delete oid 8 current snap is 15 2023-12-15T22:57:11.507 INFO:tasks.rados.rados.0.smithi071.stdout:201: expect (ObjNum 62 snap 2 seq_num 62) 2023-12-15T22:57:11.675 INFO:tasks.rados.rados.0.smithi071.stdout:200: finishing write tid 1 to smithi07183189-33 2023-12-15T22:57:11.675 INFO:tasks.rados.rados.0.smithi071.stdout:200: finishing write tid 2 to smithi07183189-33 2023-12-15T22:57:11.675 INFO:tasks.rados.rados.0.smithi071.stdout:200: finishing write tid 3 to smithi07183189-33 2023-12-15T22:57:11.675 INFO:tasks.rados.rados.0.smithi071.stdout:200: finishing write tid 4 to smithi07183189-33 2023-12-15T22:57:11.675 INFO:tasks.rados.rados.0.smithi071.stdout:200: oid 33 updating version 0 to 86 2023-12-15T22:57:11.675 INFO:tasks.rados.rados.0.smithi071.stdout:200: oid 33 version 86 is already newer than 84 2023-12-15T22:57:11.675 INFO:tasks.rados.rados.0.smithi071.stdout:200: oid 33 version 86 is already newer than 85 2023-12-15T22:57:11.676 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 33 v 86 (ObjNum 124 snap 15 seq_num 124) dirty exists 2023-12-15T22:57:11.676 INFO:tasks.rados.rados.0.smithi071.stdout:200: left oid 33 (ObjNum 124 snap 15 seq_num 124) 2023-12-15T22:57:11.676 INFO:tasks.rados.rados.0.smithi071.stdout:200: done (5 left) 2023-12-15T22:57:11.676 INFO:tasks.rados.rados.0.smithi071.stdout:201: done (4 left) 2023-12-15T22:57:11.676 INFO:tasks.rados.rados.0.smithi071.stdout:208: done (3 left) 2023-12-15T22:57:11.676 INFO:tasks.rados.rados.0.smithi071.stdout:209: write (excl) oid 15 current snap is 15 2023-12-15T22:57:11.676 INFO:tasks.rados.rados.0.smithi071.stdout:209: seq_num 128 ranges {551296=693520,2024048=127248} 2023-12-15T22:57:11.682 INFO:tasks.rados.rados.0.smithi071.stdout:209: writing smithi07183189-15 from 551296 to 1244816 tid 1 2023-12-15T22:57:11.688 INFO:tasks.rados.rados.0.smithi071.stdout:209: writing smithi07183189-15 from 2024048 to 2151296 tid 2 2023-12-15T22:57:11.688 INFO:tasks.rados.rados.0.smithi071.stdout:210: read oid 41 snap -1 2023-12-15T22:57:11.688 INFO:tasks.rados.rados.0.smithi071.stdout:210: expect (ObjNum 113 snap 10 seq_num 113) 2023-12-15T22:57:11.688 INFO:tasks.rados.rados.0.smithi071.stdout:211: delete oid 24 current snap is 15 2023-12-15T22:57:11.688 INFO:tasks.rados.rados.0.smithi071.stdout:204: expect (ObjNum 31 snap 0 seq_num 31) 2023-12-15T22:57:11.793 INFO:tasks.rados.rados.0.smithi071.stdout:206: expect (ObjNum 94 snap 7 seq_num 94) 2023-12-15T22:57:11.934 INFO:tasks.rados.rados.0.smithi071.stdout:205: finishing write tid 1 to smithi07183189-5 2023-12-15T22:57:11.934 INFO:tasks.rados.rados.0.smithi071.stdout:205: finishing write tid 2 to smithi07183189-5 2023-12-15T22:57:11.934 INFO:tasks.rados.rados.0.smithi071.stdout:205: finishing write tid 3 to smithi07183189-5 2023-12-15T22:57:11.934 INFO:tasks.rados.rados.0.smithi071.stdout:205: finishing write tid 4 to smithi07183189-5 2023-12-15T22:57:11.934 INFO:tasks.rados.rados.0.smithi071.stdout:205: oid 5 updating version 0 to 90 2023-12-15T22:57:11.934 INFO:tasks.rados.rados.0.smithi071.stdout:205: oid 5 version 90 is already newer than 89 2023-12-15T22:57:11.934 INFO:tasks.rados.rados.0.smithi071.stdout:205: oid 5 updating version 90 to 91 2023-12-15T22:57:11.934 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 5 v 91 (ObjNum 126 snap 15 seq_num 126) dirty exists 2023-12-15T22:57:11.934 INFO:tasks.rados.rados.0.smithi071.stdout:205: left oid 5 (ObjNum 126 snap 15 seq_num 126) 2023-12-15T22:57:11.934 INFO:tasks.rados.rados.0.smithi071.stdout:204: done (5 left) 2023-12-15T22:57:11.934 INFO:tasks.rados.rados.0.smithi071.stdout:205: done (4 left) 2023-12-15T22:57:11.935 INFO:tasks.rados.rados.0.smithi071.stdout:206: done (3 left) 2023-12-15T22:57:11.935 INFO:tasks.rados.rados.0.smithi071.stdout:211: done (2 left) 2023-12-15T22:57:11.935 INFO:tasks.rados.rados.0.smithi071.stdout:212: read oid 16 snap -1 2023-12-15T22:57:11.935 INFO:tasks.rados.rados.0.smithi071.stdout:212: expect (ObjNum 51 snap 0 seq_num 51) 2023-12-15T22:57:11.935 INFO:tasks.rados.rados.0.smithi071.stdout:213: write oid 37 current snap is 15 2023-12-15T22:57:11.935 INFO:tasks.rados.rados.0.smithi071.stdout:213: seq_num 130 ranges {584910=560371,1857365=327545} 2023-12-15T22:57:11.940 INFO:tasks.rados.rados.0.smithi071.stdout:213: writing smithi07183189-37 from 584910 to 1145281 tid 1 2023-12-15T22:57:11.947 INFO:tasks.rados.rados.0.smithi071.stdout:213: writing smithi07183189-37 from 1857365 to 2184910 tid 2 2023-12-15T22:57:11.947 INFO:tasks.rados.rados.0.smithi071.stdout:214: write oid 7 current snap is 15 2023-12-15T22:57:11.947 INFO:tasks.rados.rados.0.smithi071.stdout:209: finishing write tid 1 to smithi07183189-15 2023-12-15T22:57:11.947 INFO:tasks.rados.rados.0.smithi071.stdout:210: expect (ObjNum 113 snap 10 seq_num 113) 2023-12-15T22:57:12.158 INFO:tasks.rados.rados.0.smithi071.stdout:209: finishing write tid 2 to smithi07183189-15 2023-12-15T22:57:12.159 INFO:tasks.rados.rados.0.smithi071.stdout:209: finishing write tid 3 to smithi07183189-15 2023-12-15T22:57:12.159 INFO:tasks.rados.rados.0.smithi071.stdout:209: finishing write tid 4 to smithi07183189-15 2023-12-15T22:57:12.159 INFO:tasks.rados.rados.0.smithi071.stdout:209: oid 15 updating version 0 to 41 2023-12-15T22:57:12.159 INFO:tasks.rados.rados.0.smithi071.stdout:209: oid 15 version 41 is already newer than 40 2023-12-15T22:57:12.159 INFO:tasks.rados.rados.0.smithi071.stdout:209: oid 15 updating version 41 to 42 2023-12-15T22:57:12.159 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 42 (ObjNum 128 snap 15 seq_num 128) dirty exists 2023-12-15T22:57:12.159 INFO:tasks.rados.rados.0.smithi071.stdout:209: left oid 15 (ObjNum 128 snap 15 seq_num 128) 2023-12-15T22:57:12.159 INFO:tasks.rados.rados.0.smithi071.stdout:214: seq_num 131 ranges {601717=635620,1799494=402223} 2023-12-15T22:57:12.165 INFO:tasks.rados.rados.0.smithi071.stdout:214: writing smithi07183189-7 from 601717 to 1237337 tid 1 2023-12-15T22:57:12.171 INFO:tasks.rados.rados.0.smithi071.stdout:214: writing smithi07183189-7 from 1799494 to 2201717 tid 2 2023-12-15T22:57:12.171 INFO:tasks.rados.rados.0.smithi071.stdout:209: done (4 left) 2023-12-15T22:57:12.171 INFO:tasks.rados.rados.0.smithi071.stdout:210: done (3 left) 2023-12-15T22:57:12.171 INFO:tasks.rados.rados.0.smithi071.stdout:215: snap_remove snap 11 2023-12-15T22:57:12.509 INFO:tasks.rados.rados.0.smithi071.stdout:215: done (3 left) 2023-12-15T22:57:12.509 INFO:tasks.rados.rados.0.smithi071.stdout:216: delete oid 13 current snap is 15 2023-12-15T22:57:12.509 INFO:tasks.rados.rados.0.smithi071.stdout:212: expect (ObjNum 51 snap 0 seq_num 51) 2023-12-15T22:57:12.665 INFO:tasks.rados.rados.0.smithi071.stdout:213: finishing write tid 1 to smithi07183189-37 2023-12-15T22:57:12.665 INFO:tasks.rados.rados.0.smithi071.stdout:213: finishing write tid 2 to smithi07183189-37 2023-12-15T22:57:12.665 INFO:tasks.rados.rados.0.smithi071.stdout:212: done (3 left) 2023-12-15T22:57:12.665 INFO:tasks.rados.rados.0.smithi071.stdout:216: done (2 left) 2023-12-15T22:57:12.665 INFO:tasks.rados.rados.0.smithi071.stdout:217: write (excl) oid 20 current snap is 15 2023-12-15T22:57:12.665 INFO:tasks.rados.rados.0.smithi071.stdout:213: finishing write tid 3 to smithi07183189-37 2023-12-15T22:57:12.665 INFO:tasks.rados.rados.0.smithi071.stdout:217: seq_num 133 ranges {635331=786118,1916458=318873} 2023-12-15T22:57:12.672 INFO:tasks.rados.rados.0.smithi071.stdout:217: writing smithi07183189-20 from 635331 to 1421449 tid 1 2023-12-15T22:57:12.678 INFO:tasks.rados.rados.0.smithi071.stdout:217: writing smithi07183189-20 from 1916458 to 2235331 tid 2 2023-12-15T22:57:12.678 INFO:tasks.rados.rados.0.smithi071.stdout:218: delete oid 48 current snap is 15 2023-12-15T22:57:12.678 INFO:tasks.rados.rados.0.smithi071.stdout:213: finishing write tid 4 to smithi07183189-37 2023-12-15T22:57:12.678 INFO:tasks.rados.rados.0.smithi071.stdout:213: oid 37 updating version 0 to 97 2023-12-15T22:57:12.678 INFO:tasks.rados.rados.0.smithi071.stdout:213: oid 37 version 97 is already newer than 95 2023-12-15T22:57:12.678 INFO:tasks.rados.rados.0.smithi071.stdout:213: oid 37 version 97 is already newer than 97 2023-12-15T22:57:12.678 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 97 (ObjNum 130 snap 15 seq_num 130) dirty exists 2023-12-15T22:57:12.679 INFO:tasks.rados.rados.0.smithi071.stdout:213: left oid 37 (ObjNum 130 snap 15 seq_num 130) 2023-12-15T22:57:12.679 INFO:tasks.rados.rados.0.smithi071.stdout:214: finishing write tid 1 to smithi07183189-7 2023-12-15T22:57:12.679 INFO:tasks.rados.rados.0.smithi071.stdout:214: finishing write tid 2 to smithi07183189-7 2023-12-15T22:57:12.679 INFO:tasks.rados.rados.0.smithi071.stdout:214: finishing write tid 3 to smithi07183189-7 2023-12-15T22:57:12.679 INFO:tasks.rados.rados.0.smithi071.stdout:214: finishing write tid 4 to smithi07183189-7 2023-12-15T22:57:12.679 INFO:tasks.rados.rados.0.smithi071.stdout:214: oid 7 updating version 0 to 64 2023-12-15T22:57:12.679 INFO:tasks.rados.rados.0.smithi071.stdout:214: oid 7 updating version 64 to 65 2023-12-15T22:57:12.679 INFO:tasks.rados.rados.0.smithi071.stdout:214: oid 7 version 65 is already newer than 63 2023-12-15T22:57:12.679 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 65 (ObjNum 131 snap 15 seq_num 131) dirty exists 2023-12-15T22:57:12.679 INFO:tasks.rados.rados.0.smithi071.stdout:214: left oid 7 (ObjNum 131 snap 15 seq_num 131) 2023-12-15T22:57:12.716 INFO:tasks.rados.rados.0.smithi071.stdout:213: done (3 left) 2023-12-15T22:57:12.716 INFO:tasks.rados.rados.0.smithi071.stdout:214: done (2 left) 2023-12-15T22:57:12.716 INFO:tasks.rados.rados.0.smithi071.stdout:218: done (1 left) 2023-12-15T22:57:12.716 INFO:tasks.rados.rados.0.smithi071.stdout:219: read oid 16 snap -1 2023-12-15T22:57:12.716 INFO:tasks.rados.rados.0.smithi071.stdout:219: expect (ObjNum 51 snap 0 seq_num 51) 2023-12-15T22:57:12.716 INFO:tasks.rados.rados.0.smithi071.stdout:220: write (excl) oid 15 current snap is 15 2023-12-15T22:57:12.716 INFO:tasks.rados.rados.0.smithi071.stdout:220: seq_num 135 ranges {668945=536616,1633422=491363} 2023-12-15T22:57:12.723 INFO:tasks.rados.rados.0.smithi071.stdout:220: writing smithi07183189-15 from 668945 to 1205561 tid 1 2023-12-15T22:57:12.729 INFO:tasks.rados.rados.0.smithi071.stdout:220: writing smithi07183189-15 from 1633422 to 2124785 tid 2 2023-12-15T22:57:12.729 INFO:tasks.rados.rados.0.smithi071.stdout:221: snap_create 2023-12-15T22:57:12.745 INFO:tasks.rados.rados.0.smithi071.stdout:217: finishing write tid 1 to smithi07183189-20 2023-12-15T22:57:12.771 INFO:tasks.rados.rados.0.smithi071.stdout:219: expect (ObjNum 51 snap 0 seq_num 51) 2023-12-15T22:57:12.917 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:57:12.930 INFO:tasks.rados.rados.0.smithi071.stdout:220: finishing write tid 1 to smithi07183189-15 2023-12-15T22:57:12.931 INFO:tasks.rados.rados.0.smithi071.stdout:217: finishing write tid 2 to smithi07183189-20 2023-12-15T22:57:12.931 INFO:tasks.rados.rados.0.smithi071.stdout:220: finishing write tid 2 to smithi07183189-15 2023-12-15T22:57:12.931 INFO:tasks.rados.rados.0.smithi071.stdout:220: finishing write tid 3 to smithi07183189-15 2023-12-15T22:57:12.931 INFO:tasks.rados.rados.0.smithi071.stdout:217: finishing write tid 3 to smithi07183189-20 2023-12-15T22:57:12.931 INFO:tasks.rados.rados.0.smithi071.stdout:217: finishing write tid 4 to smithi07183189-20 2023-12-15T22:57:12.931 INFO:tasks.rados.rados.0.smithi071.stdout:217: oid 20 updating version 0 to 105 2023-12-15T22:57:12.931 INFO:tasks.rados.rados.0.smithi071.stdout:217: oid 20 version 105 is already newer than 103 2023-12-15T22:57:12.931 INFO:tasks.rados.rados.0.smithi071.stdout:217: oid 20 version 105 is already newer than 104 2023-12-15T22:57:12.931 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 20 v 105 (ObjNum 133 snap 15 seq_num 133) dirty exists 2023-12-15T22:57:12.931 INFO:tasks.rados.rados.0.smithi071.stdout:217: left oid 20 (ObjNum 133 snap 15 seq_num 133) 2023-12-15T22:57:12.931 INFO:tasks.rados.rados.0.smithi071.stdout:220: finishing write tid 4 to smithi07183189-15 2023-12-15T22:57:12.931 INFO:tasks.rados.rados.0.smithi071.stdout:220: oid 15 updating version 0 to 48 2023-12-15T22:57:12.931 INFO:tasks.rados.rados.0.smithi071.stdout:220: oid 15 version 48 is already newer than 48 2023-12-15T22:57:12.931 INFO:tasks.rados.rados.0.smithi071.stdout:220: oid 15 version 48 is already newer than 46 2023-12-15T22:57:12.931 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 48 (ObjNum 135 snap 15 seq_num 135) dirty exists 2023-12-15T22:57:12.931 INFO:tasks.rados.rados.0.smithi071.stdout:220: left oid 15 (ObjNum 135 snap 15 seq_num 135) 2023-12-15T22:57:13.021 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:13.019+0000 7fa376b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:13.023 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:13.024+0000 7fa376b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:13.513 INFO:tasks.rados.rados.0.smithi071.stdout:217: done (3 left) 2023-12-15T22:57:13.513 INFO:tasks.rados.rados.0.smithi071.stdout:219: done (2 left) 2023-12-15T22:57:13.513 INFO:tasks.rados.rados.0.smithi071.stdout:220: done (1 left) 2023-12-15T22:57:13.513 INFO:tasks.rados.rados.0.smithi071.stdout:221: done (0 left) 2023-12-15T22:57:13.513 INFO:tasks.rados.rados.0.smithi071.stdout:222: snap_create 2023-12-15T22:57:14.515 INFO:tasks.rados.rados.0.smithi071.stdout:222: done (0 left) 2023-12-15T22:57:14.515 INFO:tasks.rados.rados.0.smithi071.stdout:223: write (excl) oid 45 current snap is 17 2023-12-15T22:57:14.515 INFO:tasks.rados.rados.0.smithi071.stdout:223: seq_num 136 ranges {685752=611865,1975551=310201} 2023-12-15T22:57:14.522 INFO:tasks.rados.rados.0.smithi071.stdout:223: writing smithi07183189-45 from 685752 to 1297617 tid 1 2023-12-15T22:57:14.528 INFO:tasks.rados.rados.0.smithi071.stdout:223: writing smithi07183189-45 from 1975551 to 2285752 tid 2 2023-12-15T22:57:14.528 INFO:tasks.rados.rados.0.smithi071.stdout:224: write oid 25 current snap is 17 2023-12-15T22:57:14.528 INFO:tasks.rados.rados.0.smithi071.stdout:224: seq_num 137 ranges {702559=403467,1750386=495032} 2023-12-15T22:57:14.535 INFO:tasks.rados.rados.0.smithi071.stdout:224: writing smithi07183189-25 from 702559 to 1106026 tid 1 2023-12-15T22:57:14.543 INFO:tasks.rados.rados.0.smithi071.stdout:224: writing smithi07183189-25 from 1750386 to 2245418 tid 2 2023-12-15T22:57:14.543 INFO:tasks.rados.rados.0.smithi071.stdout:225: write (excl) oid 19 current snap is 17 2023-12-15T22:57:14.543 INFO:tasks.rados.rados.0.smithi071.stdout:225: seq_num 138 ranges {719366=478716,1808868=510498} 2023-12-15T22:57:14.551 INFO:tasks.rados.rados.0.smithi071.stdout:225: writing smithi07183189-19 from 719366 to 1198082 tid 1 2023-12-15T22:57:14.560 INFO:tasks.rados.rados.0.smithi071.stdout:225: writing smithi07183189-19 from 1808868 to 2319366 tid 2 2023-12-15T22:57:14.560 INFO:tasks.rados.rados.0.smithi071.stdout:226: read oid 44 snap -1 2023-12-15T22:57:14.560 INFO:tasks.rados.rados.0.smithi071.stdout:226: expect (ObjNum 105 snap 9 seq_num 105) 2023-12-15T22:57:14.560 INFO:tasks.rados.rados.0.smithi071.stdout:227: read oid 32 snap -1 2023-12-15T22:57:14.560 INFO:tasks.rados.rados.0.smithi071.stdout:227: expect (ObjNum 31 snap 0 seq_num 31) 2023-12-15T22:57:14.560 INFO:tasks.rados.rados.0.smithi071.stdout:228: read oid 36 snap -1 2023-12-15T22:57:14.560 INFO:tasks.rados.rados.0.smithi071.stdout:228: expect deleted 2023-12-15T22:57:14.560 INFO:tasks.rados.rados.0.smithi071.stdout:229: read oid 6 snap -1 2023-12-15T22:57:14.560 INFO:tasks.rados.rados.0.smithi071.stdout:229: expect (ObjNum 66 snap 3 seq_num 66) 2023-12-15T22:57:14.560 INFO:tasks.rados.rados.0.smithi071.stdout:230: snap_remove snap 15 2023-12-15T22:57:14.654 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T22:57:14.654 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T22:57:14.654 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T22:57:14.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:14.762+0000 7f3b5579e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:14.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:14.768+0000 7f3b5579e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:14.941 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:57:14.942 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-12-15T22:57:14.953 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":64,"stamp":"2023-12-15T22:57:14.519728+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":113601443,"num_objects":75,"num_object_clones":27,"num_object_copies":225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":574,"num_read_kb":274206,"num_write":592,"num_write_kb":133911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":574,"ondisk_log_size":574,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":304448,"kb_used_data":197152,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377182912,"statfs":{"total":386547056640,"available":386235301888,"internally_reserved":0,"allocated":201883648,"data_stored":201975288,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":354876,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":24,"num_read_kb":16039,"num_write":9,"num_write_kb":2638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000799"},"pg_stats":[{"pgid":"3.f","version":"17'6","reported_seq":83,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-12-15T22:57:10.545736+0000","last_change":"2023-12-15T22:57:10.527091+0000","last_active":"2023-12-15T22:57:10.545736+0000","last_peered":"2023-12-15T22:57:10.545736+0000","last_clean":"2023-12-15T22:57:10.545736+0000","last_became_active":"2023-12-15T22:56:39.503797+0000","last_became_peered":"2023-12-15T22:56:39.503797+0000","last_unstale":"2023-12-15T22:57:10.545736+0000","last_undegraded":"2023-12-15T22:57:10.545736+0000","last_fullsized":"2023-12-15T22:57:10.545736+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4537824,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":8719,"num_write":8,"num_write_kb":2181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":70,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-12-15T22:57:09.524723+0000","last_change":"2023-12-15T22:57:08.532420+0000","last_active":"2023-12-15T22:57:09.524723+0000","last_peered":"2023-12-15T22:57:09.524723+0000","last_clean":"2023-12-15T22:57:09.524723+0000","last_became_active":"2023-12-15T22:56:39.552856+0000","last_became_peered":"2023-12-15T22:56:39.552856+0000","last_unstale":"2023-12-15T22:57:09.524723+0000","last_undegraded":"2023-12-15T22:57:09.524723+0000","last_fullsized":"2023-12-15T22:57:09.524723+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"42'59","reported_seq":147,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-12-15T22:57:09.516921+0000","last_change":"2023-12-15T22:57:08.592753+0000","last_active":"2023-12-15T22:57:09.516921+0000","last_peered":"2023-12-15T22:57:09.516921+0000","last_clean":"2023-12-15T22:57:09.516921+0000","last_became_active":"2023-12-15T22:56:39.495490+0000","last_became_peered":"2023-12-15T22:56:39.495490+0000","last_unstale":"2023-12-15T22:57:09.516921+0000","last_undegraded":"2023-12-15T22:57:09.516921+0000","last_fullsized":"2023-12-15T22:57:09.516921+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13395047,"num_objects":9,"num_object_clones":3,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":66,"num_read_kb":39970,"num_write":62,"num_write_kb":17507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"43'26","reported_seq":118,"reported_epoch":44,"state":"active+clean+snaptrim","last_fresh":"2023-12-15T22:57:10.520824+0000","last_change":"2023-12-15T22:57:10.520824+0000","last_active":"2023-12-15T22:57:10.520824+0000","last_peered":"2023-12-15T22:57:10.520824+0000","last_clean":"2023-12-15T22:57:10.520824+0000","last_became_active":"2023-12-15T22:56:39.579234+0000","last_became_peered":"2023-12-15T22:56:39.579234+0000","last_unstale":"2023-12-15T22:57:10.520824+0000","last_undegraded":"2023-12-15T22:57:10.520824+0000","last_fullsized":"2023-12-15T22:57:10.520824+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7378207,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":49,"num_read_kb":37627,"num_write":31,"num_write_kb":8622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"35'20","reported_seq":89,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-12-15T22:57:10.519347+0000","last_change":"2023-12-15T22:57:10.526123+0000","last_active":"2023-12-15T22:57:10.519347+0000","last_peered":"2023-12-15T22:57:10.519347+0000","last_clean":"2023-12-15T22:57:10.519347+0000","last_became_active":"2023-12-15T22:56:39.572176+0000","last_became_peered":"2023-12-15T22:56:39.572176+0000","last_unstale":"2023-12-15T22:57:10.519347+0000","last_undegraded":"2023-12-15T22:57:10.519347+0000","last_fullsized":"2023-12-15T22:57:10.519347+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3310946,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":3685,"num_write":17,"num_write_kb":5025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"35'18","reported_seq":98,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-12-15T22:57:10.520112+0000","last_change":"2023-12-15T22:57:10.530193+0000","last_active":"2023-12-15T22:57:10.520112+0000","last_peered":"2023-12-15T22:57:10.520112+0000","last_clean":"2023-12-15T22:57:10.520112+0000","last_became_active":"2023-12-15T22:56:39.516361+0000","last_became_peered":"2023-12-15T22:56:39.516361+0000","last_unstale":"2023-12-15T22:57:10.520112+0000","last_undegraded":"2023-12-15T22:57:10.520112+0000","last_fullsized":"2023-12-15T22:57:10.520112+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6067261,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":19605,"num_write":23,"num_write_kb":6799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"44'88","reported_seq":171,"reported_epoch":44,"state":"active+clean+snaptrim","last_fresh":"2023-12-15T22:57:10.521500+0000","last_change":"2023-12-15T22:57:10.521355+0000","last_active":"2023-12-15T22:57:10.521500+0000","last_peered":"2023-12-15T22:57:10.521500+0000","last_clean":"2023-12-15T22:57:10.521500+0000","last_became_active":"2023-12-15T22:56:39.519748+0000","last_became_peered":"2023-12-15T22:56:39.519748+0000","last_unstale":"2023-12-15T22:57:10.521500+0000","last_undegraded":"2023-12-15T22:57:10.521500+0000","last_fullsized":"2023-12-15T22:57:10.521500+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18859185,"num_objects":16,"num_object_clones":7,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":80,"num_read_kb":42697,"num_write":75,"num_write_kb":22322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"37'29","reported_seq":96,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-12-15T22:57:09.505395+0000","last_change":"2023-12-15T22:57:08.583665+0000","last_active":"2023-12-15T22:57:09.505395+0000","last_peered":"2023-12-15T22:57:09.505395+0000","last_clean":"2023-12-15T22:57:09.505395+0000","last_became_active":"2023-12-15T22:56:39.543179+0000","last_became_peered":"2023-12-15T22:56:39.543179+0000","last_unstale":"2023-12-15T22:57:09.505395+0000","last_undegraded":"2023-12-15T22:57:09.505395+0000","last_fullsized":"2023-12-15T22:57:09.505395+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8016873,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":8864,"num_write":31,"num_write_kb":7951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":188,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-12-15T22:57:10.520417+0000","last_change":"2023-12-15T22:56:32.607017+0000","last_active":"2023-12-15T22:57:10.520417+0000","last_peered":"2023-12-15T22:57:10.520417+0000","last_clean":"2023-12-15T22:57:10.520417+0000","last_became_active":"2023-12-15T22:56:32.602384+0000","last_became_peered":"2023-12-15T22:56:32.602384+0000","last_unstale":"2023-12-15T22:57:10.520417+0000","last_undegraded":"2023-12-15T22:57:10.520417+0000","last_fullsized":"2023-12-15T22:57:10.520417+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_clean_scrub_stamp":"2023-12-15T22:56:31.525572+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":57,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-12-15T22:57:09.508133+0000","last_change":"2023-12-15T22:56:33.627481+0000","last_active":"2023-12-15T22:57:09.508133+0000","last_peered":"2023-12-15T22:57:09.508133+0000","last_clean":"2023-12-15T22:57:09.508133+0000","last_became_active":"2023-12-15T22:56:33.624689+0000","last_became_peered":"2023-12-15T22:56:33.624689+0000","last_unstale":"2023-12-15T22:57:09.508133+0000","last_undegraded":"2023-12-15T22:57:09.508133+0000","last_fullsized":"2023-12-15T22:57:09.508133+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"35'38","reported_seq":112,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-12-15T22:57:09.512223+0000","last_change":"2023-12-15T22:57:08.587551+0000","last_active":"2023-12-15T22:57:09.512223+0000","last_peered":"2023-12-15T22:57:09.512223+0000","last_clean":"2023-12-15T22:57:09.512223+0000","last_became_active":"2023-12-15T22:56:39.529359+0000","last_became_peered":"2023-12-15T22:56:39.529359+0000","last_unstale":"2023-12-15T22:57:09.512223+0000","last_undegraded":"2023-12-15T22:57:09.512223+0000","last_fullsized":"2023-12-15T22:57:09.512223+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7294172,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":30,"num_read_kb":10700,"num_write":40,"num_write_kb":12175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-12-15T22:57:10.521731+0000","last_change":"2023-12-15T22:56:33.655797+0000","last_active":"2023-12-15T22:57:10.521731+0000","last_peered":"2023-12-15T22:57:10.521731+0000","last_clean":"2023-12-15T22:57:10.521731+0000","last_became_active":"2023-12-15T22:56:33.652885+0000","last_became_peered":"2023-12-15T22:56:33.652885+0000","last_unstale":"2023-12-15T22:57:10.521731+0000","last_undegraded":"2023-12-15T22:57:10.521731+0000","last_fullsized":"2023-12-15T22:57:10.521731+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"37'37","reported_seq":119,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-12-15T22:57:10.528215+0000","last_change":"2023-12-15T22:57:10.549991+0000","last_active":"2023-12-15T22:57:10.528215+0000","last_peered":"2023-12-15T22:57:10.528215+0000","last_clean":"2023-12-15T22:57:10.528215+0000","last_became_active":"2023-12-15T22:56:39.543509+0000","last_became_peered":"2023-12-15T22:56:39.543509+0000","last_unstale":"2023-12-15T22:57:10.528215+0000","last_undegraded":"2023-12-15T22:57:10.528215+0000","last_fullsized":"2023-12-15T22:57:10.528215+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12504276,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":45,"num_read_kb":25376,"num_write":35,"num_write_kb":9506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":57,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-12-15T22:57:09.517824+0000","last_change":"2023-12-15T22:56:33.600485+0000","last_active":"2023-12-15T22:57:09.517824+0000","last_peered":"2023-12-15T22:57:09.517824+0000","last_clean":"2023-12-15T22:57:09.517824+0000","last_became_active":"2023-12-15T22:56:33.597320+0000","last_became_peered":"2023-12-15T22:56:33.597320+0000","last_unstale":"2023-12-15T22:57:09.517824+0000","last_undegraded":"2023-12-15T22:57:09.517824+0000","last_fullsized":"2023-12-15T22:57:09.517824+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"37'23","reported_seq":91,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-12-15T22:57:10.525447+0000","last_change":"2023-12-15T22:57:10.536161+0000","last_active":"2023-12-15T22:57:10.525447+0000","last_peered":"2023-12-15T22:57:10.525447+0000","last_clean":"2023-12-15T22:57:10.525447+0000","last_became_active":"2023-12-15T22:56:39.560980+0000","last_became_peered":"2023-12-15T22:56:39.560980+0000","last_unstale":"2023-12-15T22:57:10.525447+0000","last_undegraded":"2023-12-15T22:57:10.525447+0000","last_fullsized":"2023-12-15T22:57:10.525447+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3226911,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":13739,"num_write":17,"num_write_kb":4752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-12-15T22:57:10.522234+0000","last_change":"2023-12-15T22:56:33.648541+0000","last_active":"2023-12-15T22:57:10.522234+0000","last_peered":"2023-12-15T22:57:10.522234+0000","last_clean":"2023-12-15T22:57:10.522234+0000","last_became_active":"2023-12-15T22:56:33.643681+0000","last_became_peered":"2023-12-15T22:56:33.643681+0000","last_unstale":"2023-12-15T22:57:10.522234+0000","last_undegraded":"2023-12-15T22:57:10.522234+0000","last_fullsized":"2023-12-15T22:57:10.522234+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"35'43","reported_seq":123,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-12-15T22:57:09.525563+0000","last_change":"2023-12-15T22:57:08.529213+0000","last_active":"2023-12-15T22:57:09.525563+0000","last_peered":"2023-12-15T22:57:09.525563+0000","last_clean":"2023-12-15T22:57:09.525563+0000","last_became_active":"2023-12-15T22:56:39.506510+0000","last_became_peered":"2023-12-15T22:56:39.506510+0000","last_unstale":"2023-12-15T22:57:09.525563+0000","last_undegraded":"2023-12-15T22:57:09.525563+0000","last_fullsized":"2023-12-15T22:57:09.525563+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13563117,"num_objects":8,"num_object_clones":3,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":39,"num_read_kb":20283,"num_write":40,"num_write_kb":12077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"17'7","reported_seq":71,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-12-15T22:57:09.505419+0000","last_change":"2023-12-15T22:57:08.593066+0000","last_active":"2023-12-15T22:57:09.505419+0000","last_peered":"2023-12-15T22:57:09.505419+0000","last_clean":"2023-12-15T22:57:09.505419+0000","last_became_active":"2023-12-15T22:56:39.511159+0000","last_became_peered":"2023-12-15T22:56:39.511159+0000","last_unstale":"2023-12-15T22:57:09.505419+0000","last_undegraded":"2023-12-15T22:57:09.505419+0000","last_fullsized":"2023-12-15T22:57:09.505419+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2941192,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":9,"num_write_kb":2133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":63,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-12-15T22:57:09.530710+0000","last_change":"2023-12-15T22:56:33.633890+0000","last_active":"2023-12-15T22:57:09.530710+0000","last_peered":"2023-12-15T22:57:09.530710+0000","last_clean":"2023-12-15T22:57:09.530710+0000","last_became_active":"2023-12-15T22:56:33.631134+0000","last_became_peered":"2023-12-15T22:56:33.631134+0000","last_unstale":"2023-12-15T22:57:09.530710+0000","last_undegraded":"2023-12-15T22:57:09.530710+0000","last_fullsized":"2023-12-15T22:57:09.530710+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"42'40","reported_seq":102,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-12-15T22:57:09.531829+0000","last_change":"2023-12-15T22:57:08.618636+0000","last_active":"2023-12-15T22:57:09.531829+0000","last_peered":"2023-12-15T22:57:09.531829+0000","last_clean":"2023-12-15T22:57:09.531829+0000","last_became_active":"2023-12-15T22:56:39.532730+0000","last_became_peered":"2023-12-15T22:56:39.532730+0000","last_unstale":"2023-12-15T22:57:09.531829+0000","last_undegraded":"2023-12-15T22:57:09.531829+0000","last_fullsized":"2023-12-15T22:57:09.531829+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3764735,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":21,"num_read_kb":7996,"num_write":27,"num_write_kb":8830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":59,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-12-15T22:57:10.522980+0000","last_change":"2023-12-15T22:56:33.655736+0000","last_active":"2023-12-15T22:57:10.522980+0000","last_peered":"2023-12-15T22:57:10.522980+0000","last_clean":"2023-12-15T22:57:10.522980+0000","last_became_active":"2023-12-15T22:56:33.652917+0000","last_became_peered":"2023-12-15T22:56:33.652917+0000","last_unstale":"2023-12-15T22:57:10.522980+0000","last_undegraded":"2023-12-15T22:57:10.522980+0000","last_fullsized":"2023-12-15T22:57:10.522980+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"37'40","reported_seq":118,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-12-15T22:57:10.567141+0000","last_change":"2023-12-15T22:57:10.536038+0000","last_active":"2023-12-15T22:57:10.567141+0000","last_peered":"2023-12-15T22:57:10.567141+0000","last_clean":"2023-12-15T22:57:10.567141+0000","last_became_active":"2023-12-15T22:56:39.557252+0000","last_became_peered":"2023-12-15T22:56:39.557252+0000","last_unstale":"2023-12-15T22:57:10.567141+0000","last_undegraded":"2023-12-15T22:57:10.567141+0000","last_fullsized":"2023-12-15T22:57:10.567141+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5596698,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":27366,"num_write":36,"num_write_kb":11126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":63,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-12-15T22:57:09.523272+0000","last_change":"2023-12-15T22:56:33.632348+0000","last_active":"2023-12-15T22:57:09.523272+0000","last_peered":"2023-12-15T22:57:09.523272+0000","last_clean":"2023-12-15T22:57:09.523272+0000","last_became_active":"2023-12-15T22:56:33.629022+0000","last_became_peered":"2023-12-15T22:56:33.629022+0000","last_unstale":"2023-12-15T22:57:09.523272+0000","last_undegraded":"2023-12-15T22:57:09.523272+0000","last_fullsized":"2023-12-15T22:57:09.523272+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"17'3","reported_seq":77,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-12-15T22:57:09.529863+0000","last_change":"2023-12-15T22:57:08.532123+0000","last_active":"2023-12-15T22:57:09.529863+0000","last_peered":"2023-12-15T22:57:09.529863+0000","last_clean":"2023-12-15T22:57:09.529863+0000","last_became_active":"2023-12-15T22:56:39.535145+0000","last_became_peered":"2023-12-15T22:56:39.535145+0000","last_unstale":"2023-12-15T22:57:09.529863+0000","last_undegraded":"2023-12-15T22:57:09.529863+0000","last_fullsized":"2023-12-15T22:57:09.529863+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2554631,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":7486,"num_write":4,"num_write_kb":1011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":57,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-12-15T22:57:09.521033+0000","last_change":"2023-12-15T22:56:33.627005+0000","last_active":"2023-12-15T22:57:09.521033+0000","last_peered":"2023-12-15T22:57:09.521033+0000","last_clean":"2023-12-15T22:57:09.521033+0000","last_became_active":"2023-12-15T22:56:33.622181+0000","last_became_peered":"2023-12-15T22:56:33.622181+0000","last_unstale":"2023-12-15T22:57:09.521033+0000","last_undegraded":"2023-12-15T22:57:09.521033+0000","last_fullsized":"2023-12-15T22:57:09.521033+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":113011075,"num_objects":73,"num_object_clones":27,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":471,"num_read_kb":274115,"num_write":455,"num_write_kb":132017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":477,"ondisk_log_size":477,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705705,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":79072,"kb_used_data":52248,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94292768,"statfs":{"total":96636764160,"available":96555794432,"internally_reserved":0,"allocated":53501952,"data_stored":53515905,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705703,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":66840,"kb_used_data":40016,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94305000,"statfs":{"total":96636764160,"available":96568320000,"internally_reserved":0,"allocated":40976384,"data_stored":41069437,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705703,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":73016,"kb_used_data":46192,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94298824,"statfs":{"total":96636764160,"available":96561995776,"internally_reserved":0,"allocated":47300608,"data_stored":46857310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705704,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":85520,"kb_used_data":58696,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94286320,"statfs":{"total":96636764160,"available":96549191680,"internally_reserved":0,"allocated":60104704,"data_stored":60532636,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-15T22:57:15.091 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:57:15.194 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:15.193+0000 7fe9a0e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:15.197 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:15.197+0000 7fe9a0e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:15.522 INFO:tasks.rados.rados.0.smithi071.stdout:230: done (7 left) 2023-12-15T22:57:15.523 INFO:tasks.rados.rados.0.smithi071.stdout:231: read oid 31 snap -1 2023-12-15T22:57:15.523 INFO:tasks.rados.rados.0.smithi071.stdout:231: expect (ObjNum 94 snap 7 seq_num 94) 2023-12-15T22:57:15.523 INFO:tasks.rados.rados.0.smithi071.stdout:232: write oid 2 current snap is 17 2023-12-15T22:57:15.523 INFO:tasks.rados.rados.0.smithi071.stdout:232: seq_num 139 ranges {736173=553965,1867350=468823} 2023-12-15T22:57:15.529 INFO:tasks.rados.rados.0.smithi071.stdout:232: writing smithi07183189-2 from 736173 to 1290138 tid 1 2023-12-15T22:57:15.536 INFO:tasks.rados.rados.0.smithi071.stdout:232: writing smithi07183189-2 from 1867350 to 2336173 tid 2 2023-12-15T22:57:15.536 INFO:tasks.rados.rados.0.smithi071.stdout:233: read oid 29 snap -1 2023-12-15T22:57:15.536 INFO:tasks.rados.rados.0.smithi071.stdout:233: expect (ObjNum 71 snap 3 seq_num 71) 2023-12-15T22:57:15.536 INFO:tasks.rados.rados.0.smithi071.stdout:234: write oid 14 current snap is 17 2023-12-15T22:57:15.536 INFO:tasks.rados.rados.0.smithi071.stdout:234: seq_num 140 ranges {752980=629214,1809479=543501} 2023-12-15T22:57:15.543 INFO:tasks.rados.rados.0.smithi071.stdout:234: writing smithi07183189-14 from 752980 to 1382194 tid 1 2023-12-15T22:57:15.549 INFO:tasks.rados.rados.0.smithi071.stdout:234: writing smithi07183189-14 from 1809479 to 2352980 tid 2 2023-12-15T22:57:15.549 INFO:tasks.rados.rados.0.smithi071.stdout:235: delete oid 16 current snap is 17 2023-12-15T22:57:15.549 INFO:tasks.rados.rados.0.smithi071.stdout:226: expect (ObjNum 105 snap 9 seq_num 105) 2023-12-15T22:57:15.790 INFO:tasks.rados.rados.0.smithi071.stdout:229: expect (ObjNum 66 snap 3 seq_num 66) 2023-12-15T22:57:15.951 INFO:tasks.rados.rados.0.smithi071.stdout:225: finishing write tid 1 to smithi07183189-19 2023-12-15T22:57:15.951 INFO:tasks.rados.rados.0.smithi071.stdout:224: finishing write tid 1 to smithi07183189-25 2023-12-15T22:57:15.951 INFO:tasks.rados.rados.0.smithi071.stdout:223: finishing write tid 1 to smithi07183189-45 2023-12-15T22:57:15.951 INFO:tasks.rados.rados.0.smithi071.stdout:227: expect (ObjNum 31 snap 0 seq_num 31) 2023-12-15T22:57:16.048 INFO:tasks.rados.rados.0.smithi071.stdout:225: finishing write tid 2 to smithi07183189-19 2023-12-15T22:57:16.049 INFO:tasks.rados.rados.0.smithi071.stdout:224: finishing write tid 2 to smithi07183189-25 2023-12-15T22:57:16.049 INFO:tasks.rados.rados.0.smithi071.stdout:223: finishing write tid 2 to smithi07183189-45 2023-12-15T22:57:16.049 INFO:tasks.rados.rados.0.smithi071.stdout:224: finishing write tid 3 to smithi07183189-25 2023-12-15T22:57:16.049 INFO:tasks.rados.rados.0.smithi071.stdout:224: finishing write tid 4 to smithi07183189-25 2023-12-15T22:57:16.049 INFO:tasks.rados.rados.0.smithi071.stdout:224: oid 25 updating version 0 to 73 2023-12-15T22:57:16.049 INFO:tasks.rados.rados.0.smithi071.stdout:224: oid 25 version 73 is already newer than 71 2023-12-15T22:57:16.049 INFO:tasks.rados.rados.0.smithi071.stdout:224: oid 25 version 73 is already newer than 72 2023-12-15T22:57:16.049 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 25 v 73 (ObjNum 137 snap 17 seq_num 137) dirty exists 2023-12-15T22:57:16.049 INFO:tasks.rados.rados.0.smithi071.stdout:224: left oid 25 (ObjNum 137 snap 17 seq_num 137) 2023-12-15T22:57:16.049 INFO:tasks.rados.rados.0.smithi071.stdout:225: finishing write tid 3 to smithi07183189-19 2023-12-15T22:57:16.049 INFO:tasks.rados.rados.0.smithi071.stdout:225: finishing write tid 4 to smithi07183189-19 2023-12-15T22:57:16.050 INFO:tasks.rados.rados.0.smithi071.stdout:225: oid 19 updating version 0 to 34 2023-12-15T22:57:16.050 INFO:tasks.rados.rados.0.smithi071.stdout:225: oid 19 updating version 34 to 35 2023-12-15T22:57:16.050 INFO:tasks.rados.rados.0.smithi071.stdout:225: oid 19 version 35 is already newer than 33 2023-12-15T22:57:16.050 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 35 (ObjNum 138 snap 17 seq_num 138) dirty exists 2023-12-15T22:57:16.050 INFO:tasks.rados.rados.0.smithi071.stdout:225: left oid 19 (ObjNum 138 snap 17 seq_num 138) 2023-12-15T22:57:16.050 INFO:tasks.rados.rados.0.smithi071.stdout:223: finishing write tid 3 to smithi07183189-45 2023-12-15T22:57:16.050 INFO:tasks.rados.rados.0.smithi071.stdout:224: done (11 left) 2023-12-15T22:57:16.050 INFO:tasks.rados.rados.0.smithi071.stdout:225: done (10 left) 2023-12-15T22:57:16.050 INFO:tasks.rados.rados.0.smithi071.stdout:226: done (9 left) 2023-12-15T22:57:16.050 INFO:tasks.rados.rados.0.smithi071.stdout:227: done (8 left) 2023-12-15T22:57:16.050 INFO:tasks.rados.rados.0.smithi071.stdout:228: done (7 left) 2023-12-15T22:57:16.050 INFO:tasks.rados.rados.0.smithi071.stdout:229: done (6 left) 2023-12-15T22:57:16.050 INFO:tasks.rados.rados.0.smithi071.stdout:235: done (5 left) 2023-12-15T22:57:16.050 INFO:tasks.rados.rados.0.smithi071.stdout:236: read oid 27 snap -1 2023-12-15T22:57:16.051 INFO:tasks.rados.rados.0.smithi071.stdout:236: expect (ObjNum 74 snap 3 seq_num 74) 2023-12-15T22:57:16.051 INFO:tasks.rados.rados.0.smithi071.stdout:237: delete oid 39 current snap is 17 2023-12-15T22:57:16.051 INFO:tasks.rados.rados.0.smithi071.stdout:223: finishing write tid 4 to smithi07183189-45 2023-12-15T22:57:16.051 INFO:tasks.rados.rados.0.smithi071.stdout:223: oid 45 updating version 0 to 108 2023-12-15T22:57:16.051 INFO:tasks.rados.rados.0.smithi071.stdout:223: oid 45 updating version 108 to 109 2023-12-15T22:57:16.051 INFO:tasks.rados.rados.0.smithi071.stdout:223: oid 45 version 109 is already newer than 107 2023-12-15T22:57:16.051 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 109 (ObjNum 136 snap 17 seq_num 136) dirty exists 2023-12-15T22:57:16.051 INFO:tasks.rados.rados.0.smithi071.stdout:223: left oid 45 (ObjNum 136 snap 17 seq_num 136) 2023-12-15T22:57:16.051 INFO:tasks.rados.rados.0.smithi071.stdout:231: expect (ObjNum 94 snap 7 seq_num 94) 2023-12-15T22:57:16.191 INFO:tasks.rados.rados.0.smithi071.stdout:234: finishing write tid 1 to smithi07183189-14 2023-12-15T22:57:16.191 INFO:tasks.rados.rados.0.smithi071.stdout:233: expect (ObjNum 71 snap 3 seq_num 71) 2023-12-15T22:57:16.353 INFO:tasks.rados.rados.0.smithi071.stdout:232: finishing write tid 1 to smithi07183189-2 2023-12-15T22:57:16.353 INFO:tasks.rados.rados.0.smithi071.stdout:232: finishing write tid 2 to smithi07183189-2 2023-12-15T22:57:16.353 INFO:tasks.rados.rados.0.smithi071.stdout:234: finishing write tid 2 to smithi07183189-14 2023-12-15T22:57:16.353 INFO:tasks.rados.rados.0.smithi071.stdout:232: finishing write tid 3 to smithi07183189-2 2023-12-15T22:57:16.353 INFO:tasks.rados.rados.0.smithi071.stdout:232: finishing write tid 4 to smithi07183189-2 2023-12-15T22:57:16.353 INFO:tasks.rados.rados.0.smithi071.stdout:232: oid 2 updating version 0 to 20 2023-12-15T22:57:16.354 INFO:tasks.rados.rados.0.smithi071.stdout:232: oid 2 updating version 20 to 22 2023-12-15T22:57:16.354 INFO:tasks.rados.rados.0.smithi071.stdout:232: oid 2 version 22 is already newer than 21 2023-12-15T22:57:16.354 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 22 (ObjNum 139 snap 17 seq_num 139) dirty exists 2023-12-15T22:57:16.354 INFO:tasks.rados.rados.0.smithi071.stdout:232: left oid 2 (ObjNum 139 snap 17 seq_num 139) 2023-12-15T22:57:16.354 INFO:tasks.rados.rados.0.smithi071.stdout:234: finishing write tid 3 to smithi07183189-14 2023-12-15T22:57:16.354 INFO:tasks.rados.rados.0.smithi071.stdout:223: done (6 left) 2023-12-15T22:57:16.354 INFO:tasks.rados.rados.0.smithi071.stdout:231: done (5 left) 2023-12-15T22:57:16.354 INFO:tasks.rados.rados.0.smithi071.stdout:232: done (4 left) 2023-12-15T22:57:16.354 INFO:tasks.rados.rados.0.smithi071.stdout:233: done (3 left) 2023-12-15T22:57:16.354 INFO:tasks.rados.rados.0.smithi071.stdout:237: done (2 left) 2023-12-15T22:57:16.354 INFO:tasks.rados.rados.0.smithi071.stdout:238: snap_remove snap 12 2023-12-15T22:57:16.523 INFO:tasks.rados.rados.0.smithi071.stdout:238: done (2 left) 2023-12-15T22:57:16.524 INFO:tasks.rados.rados.0.smithi071.stdout:239: read oid 26 snap -1 2023-12-15T22:57:16.524 INFO:tasks.rados.rados.0.smithi071.stdout:239: expect (ObjNum 120 snap 10 seq_num 120) 2023-12-15T22:57:16.524 INFO:tasks.rados.rados.0.smithi071.stdout:240: snap_remove snap 16 2023-12-15T22:57:17.266 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:57:17.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:17.373+0000 7f6c7579e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:17.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:17.378+0000 7f6c7579e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:17.530 INFO:tasks.rados.rados.0.smithi071.stdout:240: done (3 left) 2023-12-15T22:57:17.530 INFO:tasks.rados.rados.0.smithi071.stdout:241: read oid 25 snap -1 2023-12-15T22:57:17.530 INFO:tasks.rados.rados.0.smithi071.stdout:241: expect (ObjNum 137 snap 17 seq_num 137) 2023-12-15T22:57:17.530 INFO:tasks.rados.rados.0.smithi071.stdout:242: read oid 3 snap -1 2023-12-15T22:57:17.530 INFO:tasks.rados.rados.0.smithi071.stdout:242: expect (ObjNum 119 snap 10 seq_num 119) 2023-12-15T22:57:17.531 INFO:tasks.rados.rados.0.smithi071.stdout:243: snap_create 2023-12-15T22:57:17.531 INFO:tasks.rados.rados.0.smithi071.stdout:234: finishing write tid 4 to smithi07183189-14 2023-12-15T22:57:17.531 INFO:tasks.rados.rados.0.smithi071.stdout:234: oid 14 updating version 0 to 46 2023-12-15T22:57:17.531 INFO:tasks.rados.rados.0.smithi071.stdout:234: oid 14 version 46 is already newer than 44 2023-12-15T22:57:17.531 INFO:tasks.rados.rados.0.smithi071.stdout:234: oid 14 version 46 is already newer than 45 2023-12-15T22:57:17.531 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 46 (ObjNum 140 snap 17 seq_num 140) dirty exists 2023-12-15T22:57:17.531 INFO:tasks.rados.rados.0.smithi071.stdout:234: left oid 14 (ObjNum 140 snap 17 seq_num 140) 2023-12-15T22:57:17.531 INFO:tasks.rados.rados.0.smithi071.stdout:236: expect (ObjNum 74 snap 3 seq_num 74) 2023-12-15T22:57:17.694 INFO:tasks.rados.rados.0.smithi071.stdout:239: expect (ObjNum 120 snap 10 seq_num 120) 2023-12-15T22:57:17.840 INFO:tasks.rados.rados.0.smithi071.stdout:242: expect (ObjNum 119 snap 10 seq_num 119) 2023-12-15T22:57:17.941 INFO:tasks.rados.rados.0.smithi071.stdout:241: expect (ObjNum 137 snap 17 seq_num 137) 2023-12-15T22:57:18.534 INFO:tasks.rados.rados.0.smithi071.stdout:234: done (5 left) 2023-12-15T22:57:18.534 INFO:tasks.rados.rados.0.smithi071.stdout:236: done (4 left) 2023-12-15T22:57:18.534 INFO:tasks.rados.rados.0.smithi071.stdout:239: done (3 left) 2023-12-15T22:57:18.534 INFO:tasks.rados.rados.0.smithi071.stdout:241: done (2 left) 2023-12-15T22:57:18.534 INFO:tasks.rados.rados.0.smithi071.stdout:242: done (1 left) 2023-12-15T22:57:18.534 INFO:tasks.rados.rados.0.smithi071.stdout:243: done (0 left) 2023-12-15T22:57:18.534 INFO:tasks.rados.rados.0.smithi071.stdout:244: write (excl) oid 49 current snap is 18 2023-12-15T22:57:18.534 INFO:tasks.rados.rados.0.smithi071.stdout:244: seq_num 143 ranges {403401=454961,1584925=506039} 2023-12-15T22:57:18.538 INFO:tasks.rados.rados.0.smithi071.stdout:244: writing smithi07183189-49 from 403401 to 858362 tid 1 2023-12-15T22:57:18.545 INFO:tasks.rados.rados.0.smithi071.stdout:244: writing smithi07183189-49 from 1584925 to 2090964 tid 2 2023-12-15T22:57:18.545 INFO:tasks.rados.rados.0.smithi071.stdout:245: snap_create 2023-12-15T22:57:18.628 INFO:tasks.rados.rados.0.smithi071.stdout:244: finishing write tid 1 to smithi07183189-49 2023-12-15T22:57:18.649 INFO:tasks.rados.rados.0.smithi071.stdout:244: finishing write tid 2 to smithi07183189-49 2023-12-15T22:57:18.663 INFO:tasks.rados.rados.0.smithi071.stdout:244: finishing write tid 3 to smithi07183189-49 2023-12-15T22:57:18.665 INFO:tasks.rados.rados.0.smithi071.stdout:244: finishing write tid 4 to smithi07183189-49 2023-12-15T22:57:18.666 INFO:tasks.rados.rados.0.smithi071.stdout:244: oid 49 updating version 0 to 49 2023-12-15T22:57:18.666 INFO:tasks.rados.rados.0.smithi071.stdout:244: oid 49 updating version 49 to 50 2023-12-15T22:57:18.666 INFO:tasks.rados.rados.0.smithi071.stdout:244: oid 49 updating version 50 to 53 2023-12-15T22:57:18.666 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 49 v 53 (ObjNum 143 snap 18 seq_num 143) dirty exists 2023-12-15T22:57:18.666 INFO:tasks.rados.rados.0.smithi071.stdout:244: left oid 49 (ObjNum 143 snap 18 seq_num 143) 2023-12-15T22:57:19.445 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:57:19.537 INFO:tasks.rados.rados.0.smithi071.stdout:244: done (1 left) 2023-12-15T22:57:19.537 INFO:tasks.rados.rados.0.smithi071.stdout:245: done (0 left) 2023-12-15T22:57:19.537 INFO:tasks.rados.rados.0.smithi071.stdout:246: read oid 49 snap -1 2023-12-15T22:57:19.537 INFO:tasks.rados.rados.0.smithi071.stdout:246: expect (ObjNum 143 snap 18 seq_num 143) 2023-12-15T22:57:19.537 INFO:tasks.rados.rados.0.smithi071.stdout:247: write oid 37 current snap is 19 2023-12-15T22:57:19.537 INFO:tasks.rados.rados.0.smithi071.stdout:247: seq_num 144 ranges {420208=530210,1527054=766050} 2023-12-15T22:57:19.542 INFO:tasks.rados.rados.0.smithi071.stdout:247: writing smithi07183189-37 from 420208 to 950418 tid 1 2023-12-15T22:57:19.551 INFO:tasks.rados.rados.0.smithi071.stdout:247: writing smithi07183189-37 from 1527054 to 2293104 tid 2 2023-12-15T22:57:19.551 INFO:tasks.rados.rados.0.smithi071.stdout:248: write oid 48 current snap is 19 2023-12-15T22:57:19.551 INFO:tasks.rados.rados.0.smithi071.stdout:248: seq_num 145 ranges {437015=721812,1701889=509708} 2023-12-15T22:57:19.559 INFO:tasks.rados.rados.0.smithi071.stdout:248: writing smithi07183189-48 from 437015 to 1158827 tid 1 2023-12-15T22:57:19.565 INFO:tasks.rados.rados.0.smithi071.stdout:248: writing smithi07183189-48 from 1701889 to 2211597 tid 2 2023-12-15T22:57:19.565 INFO:tasks.rados.rados.0.smithi071.stdout:249: snap_create 2023-12-15T22:57:19.583 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:19.581+0000 7f8d4559e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:19.587 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:19.587+0000 7f8d4559e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:19.631 INFO:tasks.rados.rados.0.smithi071.stdout:246: expect (ObjNum 143 snap 18 seq_num 143) 2023-12-15T22:57:19.786 INFO:tasks.rados.rados.0.smithi071.stdout:248: finishing write tid 1 to smithi07183189-48 2023-12-15T22:57:19.786 INFO:tasks.rados.rados.0.smithi071.stdout:247: finishing write tid 1 to smithi07183189-37 2023-12-15T22:57:19.786 INFO:tasks.rados.rados.0.smithi071.stdout:248: finishing write tid 2 to smithi07183189-48 2023-12-15T22:57:19.786 INFO:tasks.rados.rados.0.smithi071.stdout:248: finishing write tid 3 to smithi07183189-48 2023-12-15T22:57:19.786 INFO:tasks.rados.rados.0.smithi071.stdout:248: finishing write tid 4 to smithi07183189-48 2023-12-15T22:57:19.787 INFO:tasks.rados.rados.0.smithi071.stdout:248: oid 48 updating version 0 to 52 2023-12-15T22:57:19.787 INFO:tasks.rados.rados.0.smithi071.stdout:248: oid 48 version 52 is already newer than 51 2023-12-15T22:57:19.787 INFO:tasks.rados.rados.0.smithi071.stdout:248: oid 48 updating version 52 to 53 2023-12-15T22:57:19.787 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 53 (ObjNum 145 snap 19 seq_num 145) dirty exists 2023-12-15T22:57:19.787 INFO:tasks.rados.rados.0.smithi071.stdout:248: left oid 48 (ObjNum 145 snap 19 seq_num 145) 2023-12-15T22:57:19.787 INFO:tasks.rados.rados.0.smithi071.stdout:247: finishing write tid 2 to smithi07183189-37 2023-12-15T22:57:19.787 INFO:tasks.rados.rados.0.smithi071.stdout:247: finishing write tid 3 to smithi07183189-37 2023-12-15T22:57:19.787 INFO:tasks.rados.rados.0.smithi071.stdout:247: finishing write tid 4 to smithi07183189-37 2023-12-15T22:57:19.787 INFO:tasks.rados.rados.0.smithi071.stdout:247: oid 37 updating version 0 to 125 2023-12-15T22:57:19.787 INFO:tasks.rados.rados.0.smithi071.stdout:247: oid 37 version 125 is already newer than 123 2023-12-15T22:57:19.787 INFO:tasks.rados.rados.0.smithi071.stdout:247: oid 37 version 125 is already newer than 124 2023-12-15T22:57:19.787 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 125 (ObjNum 144 snap 19 seq_num 144) dirty exists 2023-12-15T22:57:19.787 INFO:tasks.rados.rados.0.smithi071.stdout:247: left oid 37 (ObjNum 144 snap 19 seq_num 144) 2023-12-15T22:57:19.957 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T22:57:19.957 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T22:57:19.958 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T22:57:20.063 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:20.060+0000 7fcc4919e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:20.066 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:20.066+0000 7fcc4919e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:20.239 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:57:20.239 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-12-15T22:57:20.251 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":71,"stamp":"2023-12-15T22:57:19.533852+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":105683714,"num_objects":83,"num_object_clones":35,"num_object_copies":249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":620,"num_read_kb":314919,"num_write":624,"num_write_kb":141647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":637,"ondisk_log_size":637,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":326808,"kb_used_data":217664,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377160552,"statfs":{"total":386547056640,"available":386212405248,"internally_reserved":0,"allocated":222887936,"data_stored":227081613,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2505974,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":22341,"num_write":20,"num_write_kb":4914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.012595"},"pg_stats":[{"pgid":"3.f","version":"17'6","reported_seq":96,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-15T22:57:14.636062+0000","last_change":"2023-12-15T22:57:12.526116+0000","last_active":"2023-12-15T22:57:14.636062+0000","last_peered":"2023-12-15T22:57:14.636062+0000","last_clean":"2023-12-15T22:57:14.636062+0000","last_became_active":"2023-12-15T22:56:39.503797+0000","last_became_peered":"2023-12-15T22:56:39.503797+0000","last_unstale":"2023-12-15T22:57:14.636062+0000","last_undegraded":"2023-12-15T22:57:14.636062+0000","last_fullsized":"2023-12-15T22:57:14.636062+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4537824,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":16105,"num_write":8,"num_write_kb":2181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":83,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-12-15T22:57:14.524107+0000","last_change":"2023-12-15T22:57:12.628374+0000","last_active":"2023-12-15T22:57:14.524107+0000","last_peered":"2023-12-15T22:57:14.524107+0000","last_clean":"2023-12-15T22:57:14.524107+0000","last_became_active":"2023-12-15T22:56:39.552856+0000","last_became_peered":"2023-12-15T22:56:39.552856+0000","last_unstale":"2023-12-15T22:57:14.524107+0000","last_undegraded":"2023-12-15T22:57:14.524107+0000","last_fullsized":"2023-12-15T22:57:14.524107+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"46'69","reported_seq":168,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-12-15T22:57:13.619705+0000","last_change":"2023-12-15T22:57:12.792815+0000","last_active":"2023-12-15T22:57:13.619705+0000","last_peered":"2023-12-15T22:57:13.619705+0000","last_clean":"2023-12-15T22:57:13.619705+0000","last_became_active":"2023-12-15T22:56:39.495490+0000","last_became_peered":"2023-12-15T22:56:39.495490+0000","last_unstale":"2023-12-15T22:57:13.619705+0000","last_undegraded":"2023-12-15T22:57:13.619705+0000","last_fullsized":"2023-12-15T22:57:13.619705+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12470695,"num_objects":10,"num_object_clones":4,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":79,"num_read_kb":54994,"num_write":66,"num_write_kb":18521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"48'35","reported_seq":135,"reported_epoch":49,"state":"active+clean+snaptrim","last_fresh":"2023-12-15T22:57:15.557786+0000","last_change":"2023-12-15T22:57:15.557786+0000","last_active":"2023-12-15T22:57:15.557786+0000","last_peered":"2023-12-15T22:57:15.557786+0000","last_clean":"2023-12-15T22:57:15.557786+0000","last_became_active":"2023-12-15T22:56:39.579234+0000","last_became_peered":"2023-12-15T22:56:39.579234+0000","last_unstale":"2023-12-15T22:57:15.557786+0000","last_undegraded":"2023-12-15T22:57:15.557786+0000","last_fullsized":"2023-12-15T22:57:15.557786+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9697573,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":51,"num_read_kb":37628,"num_write":35,"num_write_kb":9589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"35'20","reported_seq":100,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-15T22:57:15.552067+0000","last_change":"2023-12-15T22:57:15.554470+0000","last_active":"2023-12-15T22:57:15.552067+0000","last_peered":"2023-12-15T22:57:15.552067+0000","last_clean":"2023-12-15T22:57:15.552067+0000","last_became_active":"2023-12-15T22:56:39.572176+0000","last_became_peered":"2023-12-15T22:56:39.572176+0000","last_unstale":"2023-12-15T22:57:15.552067+0000","last_undegraded":"2023-12-15T22:57:15.552067+0000","last_fullsized":"2023-12-15T22:57:15.552067+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3310946,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":3685,"num_write":17,"num_write_kb":5025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"35'18","reported_seq":110,"reported_epoch":49,"state":"active+clean+snaptrim","last_fresh":"2023-12-15T22:57:15.564200+0000","last_change":"2023-12-15T22:57:15.564200+0000","last_active":"2023-12-15T22:57:15.564200+0000","last_peered":"2023-12-15T22:57:15.564200+0000","last_clean":"2023-12-15T22:57:15.564200+0000","last_became_active":"2023-12-15T22:56:39.516361+0000","last_became_peered":"2023-12-15T22:56:39.516361+0000","last_unstale":"2023-12-15T22:57:15.564200+0000","last_undegraded":"2023-12-15T22:57:15.564200+0000","last_fullsized":"2023-12-15T22:57:15.564200+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6067261,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":19605,"num_write":23,"num_write_kb":6799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"48'109","reported_seq":199,"reported_epoch":49,"state":"active+clean+snaptrim","last_fresh":"2023-12-15T22:57:15.560652+0000","last_change":"2023-12-15T22:57:15.560652+0000","last_active":"2023-12-15T22:57:15.560652+0000","last_peered":"2023-12-15T22:57:15.560652+0000","last_clean":"2023-12-15T22:57:15.560652+0000","last_became_active":"2023-12-15T22:56:39.519748+0000","last_became_peered":"2023-12-15T22:56:39.519748+0000","last_unstale":"2023-12-15T22:57:15.560652+0000","last_undegraded":"2023-12-15T22:57:15.560652+0000","last_fullsized":"2023-12-15T22:57:15.560652+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16151494,"num_objects":19,"num_object_clones":10,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":86,"num_read_kb":42701,"num_write":90,"num_write_kb":25654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"37'29","reported_seq":109,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-12-15T22:57:14.553769+0000","last_change":"2023-12-15T22:57:12.701564+0000","last_active":"2023-12-15T22:57:14.553769+0000","last_peered":"2023-12-15T22:57:14.553769+0000","last_clean":"2023-12-15T22:57:14.553769+0000","last_became_active":"2023-12-15T22:56:39.543179+0000","last_became_peered":"2023-12-15T22:56:39.543179+0000","last_unstale":"2023-12-15T22:57:14.553769+0000","last_undegraded":"2023-12-15T22:57:14.553769+0000","last_fullsized":"2023-12-15T22:57:14.553769+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8016873,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":8864,"num_write":31,"num_write_kb":7951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":196,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-15T22:57:15.547708+0000","last_change":"2023-12-15T22:56:32.607017+0000","last_active":"2023-12-15T22:57:15.547708+0000","last_peered":"2023-12-15T22:57:15.547708+0000","last_clean":"2023-12-15T22:57:15.547708+0000","last_became_active":"2023-12-15T22:56:32.602384+0000","last_became_peered":"2023-12-15T22:56:32.602384+0000","last_unstale":"2023-12-15T22:57:15.547708+0000","last_undegraded":"2023-12-15T22:57:15.547708+0000","last_fullsized":"2023-12-15T22:57:15.547708+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_clean_scrub_stamp":"2023-12-15T22:56:31.525572+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":67,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-12-15T22:57:14.562551+0000","last_change":"2023-12-15T22:56:33.627481+0000","last_active":"2023-12-15T22:57:14.562551+0000","last_peered":"2023-12-15T22:57:14.562551+0000","last_clean":"2023-12-15T22:57:14.562551+0000","last_became_active":"2023-12-15T22:56:33.624689+0000","last_became_peered":"2023-12-15T22:56:33.624689+0000","last_unstale":"2023-12-15T22:57:14.562551+0000","last_undegraded":"2023-12-15T22:57:14.562551+0000","last_fullsized":"2023-12-15T22:57:14.562551+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"46'48","reported_seq":131,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-12-15T22:57:13.620221+0000","last_change":"2023-12-15T22:57:12.745218+0000","last_active":"2023-12-15T22:57:13.620221+0000","last_peered":"2023-12-15T22:57:13.620221+0000","last_clean":"2023-12-15T22:57:13.620221+0000","last_became_active":"2023-12-15T22:56:39.529359+0000","last_became_peered":"2023-12-15T22:56:39.529359+0000","last_unstale":"2023-12-15T22:57:13.620221+0000","last_undegraded":"2023-12-15T22:57:13.620221+0000","last_fullsized":"2023-12-15T22:57:13.620221+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5747961,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":34,"num_read_kb":10702,"num_write":48,"num_write_kb":13983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":71,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-15T22:57:14.539105+0000","last_change":"2023-12-15T22:56:33.655797+0000","last_active":"2023-12-15T22:57:14.539105+0000","last_peered":"2023-12-15T22:57:14.539105+0000","last_clean":"2023-12-15T22:57:14.539105+0000","last_became_active":"2023-12-15T22:56:33.652885+0000","last_became_peered":"2023-12-15T22:56:33.652885+0000","last_unstale":"2023-12-15T22:57:14.539105+0000","last_undegraded":"2023-12-15T22:57:14.539105+0000","last_fullsized":"2023-12-15T22:57:14.539105+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"46'42","reported_seq":135,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-15T22:57:15.556386+0000","last_change":"2023-12-15T22:57:15.556386+0000","last_active":"2023-12-15T22:57:15.556386+0000","last_peered":"2023-12-15T22:57:15.556386+0000","last_clean":"2023-12-15T22:57:15.556386+0000","last_became_active":"2023-12-15T22:56:39.543509+0000","last_became_peered":"2023-12-15T22:56:39.543509+0000","last_unstale":"2023-12-15T22:57:15.556386+0000","last_undegraded":"2023-12-15T22:57:15.556386+0000","last_fullsized":"2023-12-15T22:57:15.556386+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10386627,"num_objects":9,"num_object_clones":4,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":51,"num_read_kb":33776,"num_write":36,"num_write_kb":10121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":67,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-12-15T22:57:14.550382+0000","last_change":"2023-12-15T22:56:33.600485+0000","last_active":"2023-12-15T22:57:14.550382+0000","last_peered":"2023-12-15T22:57:14.550382+0000","last_clean":"2023-12-15T22:57:14.550382+0000","last_became_active":"2023-12-15T22:56:33.597320+0000","last_became_peered":"2023-12-15T22:56:33.597320+0000","last_unstale":"2023-12-15T22:57:14.550382+0000","last_undegraded":"2023-12-15T22:57:14.550382+0000","last_fullsized":"2023-12-15T22:57:14.550382+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"37'23","reported_seq":102,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-15T22:57:15.545779+0000","last_change":"2023-12-15T22:57:15.553202+0000","last_active":"2023-12-15T22:57:15.545779+0000","last_peered":"2023-12-15T22:57:15.545779+0000","last_clean":"2023-12-15T22:57:15.545779+0000","last_became_active":"2023-12-15T22:56:39.560980+0000","last_became_peered":"2023-12-15T22:56:39.560980+0000","last_unstale":"2023-12-15T22:57:15.545779+0000","last_undegraded":"2023-12-15T22:57:15.545779+0000","last_fullsized":"2023-12-15T22:57:15.545779+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3226911,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":13739,"num_write":17,"num_write_kb":4752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":73,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-15T22:57:15.561423+0000","last_change":"2023-12-15T22:56:33.648541+0000","last_active":"2023-12-15T22:57:15.561423+0000","last_peered":"2023-12-15T22:57:15.561423+0000","last_clean":"2023-12-15T22:57:15.561423+0000","last_became_active":"2023-12-15T22:56:33.643681+0000","last_became_peered":"2023-12-15T22:56:33.643681+0000","last_unstale":"2023-12-15T22:57:15.561423+0000","last_undegraded":"2023-12-15T22:57:15.561423+0000","last_fullsized":"2023-12-15T22:57:15.561423+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"35'43","reported_seq":136,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-12-15T22:57:14.522831+0000","last_change":"2023-12-15T22:57:12.627125+0000","last_active":"2023-12-15T22:57:14.522831+0000","last_peered":"2023-12-15T22:57:14.522831+0000","last_clean":"2023-12-15T22:57:14.522831+0000","last_became_active":"2023-12-15T22:56:39.506510+0000","last_became_peered":"2023-12-15T22:56:39.506510+0000","last_unstale":"2023-12-15T22:57:14.522831+0000","last_undegraded":"2023-12-15T22:57:14.522831+0000","last_fullsized":"2023-12-15T22:57:14.522831+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13563117,"num_objects":8,"num_object_clones":3,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":39,"num_read_kb":20283,"num_write":40,"num_write_kb":12077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"46'11","reported_seq":83,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-12-15T22:57:13.623208+0000","last_change":"2023-12-15T22:57:12.763621+0000","last_active":"2023-12-15T22:57:13.623208+0000","last_peered":"2023-12-15T22:57:13.623208+0000","last_clean":"2023-12-15T22:57:13.623208+0000","last_became_active":"2023-12-15T22:56:39.511159+0000","last_became_peered":"2023-12-15T22:56:39.511159+0000","last_unstale":"2023-12-15T22:57:13.623208+0000","last_undegraded":"2023-12-15T22:57:13.623208+0000","last_fullsized":"2023-12-15T22:57:13.623208+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":5,"num_read_kb":2,"num_write":9,"num_write_kb":2133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":73,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-12-15T22:57:14.529874+0000","last_change":"2023-12-15T22:56:33.633890+0000","last_active":"2023-12-15T22:57:14.529874+0000","last_peered":"2023-12-15T22:57:14.529874+0000","last_clean":"2023-12-15T22:57:14.529874+0000","last_became_active":"2023-12-15T22:56:33.631134+0000","last_became_peered":"2023-12-15T22:56:33.631134+0000","last_unstale":"2023-12-15T22:57:14.529874+0000","last_undegraded":"2023-12-15T22:57:14.529874+0000","last_fullsized":"2023-12-15T22:57:14.529874+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"46'44","reported_seq":117,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-12-15T22:57:14.530883+0000","last_change":"2023-12-15T22:57:12.637328+0000","last_active":"2023-12-15T22:57:14.530883+0000","last_peered":"2023-12-15T22:57:14.530883+0000","last_clean":"2023-12-15T22:57:14.530883+0000","last_became_active":"2023-12-15T22:56:39.532730+0000","last_became_peered":"2023-12-15T22:56:39.532730+0000","last_unstale":"2023-12-15T22:57:14.530883+0000","last_undegraded":"2023-12-15T22:57:14.530883+0000","last_fullsized":"2023-12-15T22:57:14.530883+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3764735,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":23,"num_read_kb":11342,"num_write":27,"num_write_kb":8830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":67,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-15T22:57:15.547353+0000","last_change":"2023-12-15T22:56:33.655736+0000","last_active":"2023-12-15T22:57:15.547353+0000","last_peered":"2023-12-15T22:57:15.547353+0000","last_clean":"2023-12-15T22:57:15.547353+0000","last_became_active":"2023-12-15T22:56:33.652917+0000","last_became_peered":"2023-12-15T22:56:33.652917+0000","last_unstale":"2023-12-15T22:57:15.547353+0000","last_undegraded":"2023-12-15T22:57:15.547353+0000","last_fullsized":"2023-12-15T22:57:15.547353+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"37'40","reported_seq":132,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-15T22:57:15.564996+0000","last_change":"2023-12-15T22:57:15.553805+0000","last_active":"2023-12-15T22:57:15.564996+0000","last_peered":"2023-12-15T22:57:15.564996+0000","last_clean":"2023-12-15T22:57:15.564996+0000","last_became_active":"2023-12-15T22:56:39.557252+0000","last_became_peered":"2023-12-15T22:56:39.557252+0000","last_unstale":"2023-12-15T22:57:15.564996+0000","last_undegraded":"2023-12-15T22:57:15.564996+0000","last_fullsized":"2023-12-15T22:57:15.564996+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5596698,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":45,"num_read_kb":33916,"num_write":36,"num_write_kb":11126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":73,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-12-15T22:57:14.529835+0000","last_change":"2023-12-15T22:56:33.632348+0000","last_active":"2023-12-15T22:57:14.529835+0000","last_peered":"2023-12-15T22:57:14.529835+0000","last_clean":"2023-12-15T22:57:14.529835+0000","last_became_active":"2023-12-15T22:56:33.629022+0000","last_became_peered":"2023-12-15T22:56:33.629022+0000","last_unstale":"2023-12-15T22:57:14.529835+0000","last_undegraded":"2023-12-15T22:57:14.529835+0000","last_fullsized":"2023-12-15T22:57:14.529835+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"17'3","reported_seq":90,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-12-15T22:57:14.521791+0000","last_change":"2023-12-15T22:57:12.627003+0000","last_active":"2023-12-15T22:57:14.521791+0000","last_peered":"2023-12-15T22:57:14.521791+0000","last_clean":"2023-12-15T22:57:14.521791+0000","last_became_active":"2023-12-15T22:56:39.535145+0000","last_became_peered":"2023-12-15T22:56:39.535145+0000","last_unstale":"2023-12-15T22:57:14.521791+0000","last_undegraded":"2023-12-15T22:57:14.521791+0000","last_fullsized":"2023-12-15T22:57:14.521791+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2554631,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":7486,"num_write":4,"num_write_kb":1011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":67,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-12-15T22:57:14.555609+0000","last_change":"2023-12-15T22:56:33.627005+0000","last_active":"2023-12-15T22:57:14.555609+0000","last_peered":"2023-12-15T22:57:14.555609+0000","last_clean":"2023-12-15T22:57:14.555609+0000","last_became_active":"2023-12-15T22:56:33.622181+0000","last_became_peered":"2023-12-15T22:56:33.622181+0000","last_unstale":"2023-12-15T22:57:14.555609+0000","last_undegraded":"2023-12-15T22:57:14.555609+0000","last_fullsized":"2023-12-15T22:57:14.555609+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":105093346,"num_objects":81,"num_object_clones":35,"num_object_copies":243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":517,"num_read_kb":314828,"num_write":487,"num_write_kb":139753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":540,"ondisk_log_size":540,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705709,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":81924,"kb_used_data":54484,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94289916,"statfs":{"total":96636764160,"available":96552873984,"internally_reserved":0,"allocated":55791616,"data_stored":56414621,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705708,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":71960,"kb_used_data":45136,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94299880,"statfs":{"total":96636764160,"available":96563077120,"internally_reserved":0,"allocated":46219264,"data_stored":46682040,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705708,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":80968,"kb_used_data":53528,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94290872,"statfs":{"total":96636764160,"available":96553852928,"internally_reserved":0,"allocated":54812672,"data_stored":55352848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705709,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":91956,"kb_used_data":64516,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94279884,"statfs":{"total":96636764160,"available":96542601216,"internally_reserved":0,"allocated":66064384,"data_stored":68632104,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-15T22:57:20.542 INFO:tasks.rados.rados.0.smithi071.stdout:246: done (3 left) 2023-12-15T22:57:20.542 INFO:tasks.rados.rados.0.smithi071.stdout:247: done (2 left) 2023-12-15T22:57:20.543 INFO:tasks.rados.rados.0.smithi071.stdout:248: done (1 left) 2023-12-15T22:57:20.543 INFO:tasks.rados.rados.0.smithi071.stdout:249: done (0 left) 2023-12-15T22:57:20.543 INFO:tasks.rados.rados.0.smithi071.stdout:250: read oid 44 snap -1 2023-12-15T22:57:20.543 INFO:tasks.rados.rados.0.smithi071.stdout:250: expect (ObjNum 105 snap 9 seq_num 105) 2023-12-15T22:57:20.543 INFO:tasks.rados.rados.0.smithi071.stdout:251: write (excl) oid 32 current snap is 20 2023-12-15T22:57:20.543 INFO:tasks.rados.rados.0.smithi071.stdout:251: seq_num 146 ranges {453822=797061,1760371=653366} 2023-12-15T22:57:20.548 INFO:tasks.rados.rados.0.smithi071.stdout:251: writing smithi07183189-32 from 453822 to 1250883 tid 1 2023-12-15T22:57:20.556 INFO:tasks.rados.rados.0.smithi071.stdout:251: writing smithi07183189-32 from 1760371 to 2413737 tid 2 2023-12-15T22:57:20.556 INFO:tasks.rados.rados.0.smithi071.stdout:252: write oid 10 current snap is 20 2023-12-15T22:57:20.556 INFO:tasks.rados.rados.0.smithi071.stdout:252: seq_num 147 ranges {470629=472310,1418853=513377} 2023-12-15T22:57:20.562 INFO:tasks.rados.rados.0.smithi071.stdout:252: writing smithi07183189-10 from 470629 to 942939 tid 1 2023-12-15T22:57:20.569 INFO:tasks.rados.rados.0.smithi071.stdout:252: writing smithi07183189-10 from 1418853 to 1932230 tid 2 2023-12-15T22:57:20.570 INFO:tasks.rados.rados.0.smithi071.stdout:253: delete oid 46 current snap is 20 2023-12-15T22:57:20.582 INFO:tasks.rados.rados.0.smithi071.stdout:250: expect (ObjNum 105 snap 9 seq_num 105) 2023-12-15T22:57:20.819 INFO:tasks.rados.rados.0.smithi071.stdout:251: finishing write tid 1 to smithi07183189-32 2023-12-15T22:57:20.819 INFO:tasks.rados.rados.0.smithi071.stdout:252: finishing write tid 1 to smithi07183189-10 2023-12-15T22:57:20.819 INFO:tasks.rados.rados.0.smithi071.stdout:251: finishing write tid 2 to smithi07183189-32 2023-12-15T22:57:20.819 INFO:tasks.rados.rados.0.smithi071.stdout:252: finishing write tid 2 to smithi07183189-10 2023-12-15T22:57:20.819 INFO:tasks.rados.rados.0.smithi071.stdout:251: finishing write tid 3 to smithi07183189-32 2023-12-15T22:57:20.819 INFO:tasks.rados.rados.0.smithi071.stdout:251: finishing write tid 4 to smithi07183189-32 2023-12-15T22:57:20.820 INFO:tasks.rados.rados.0.smithi071.stdout:251: oid 32 updating version 0 to 8 2023-12-15T22:57:20.820 INFO:tasks.rados.rados.0.smithi071.stdout:251: oid 32 updating version 8 to 9 2023-12-15T22:57:20.820 INFO:tasks.rados.rados.0.smithi071.stdout:251: oid 32 updating version 9 to 10 2023-12-15T22:57:20.820 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 10 (ObjNum 146 snap 20 seq_num 146) dirty exists 2023-12-15T22:57:20.820 INFO:tasks.rados.rados.0.smithi071.stdout:251: left oid 32 (ObjNum 146 snap 20 seq_num 146) 2023-12-15T22:57:20.820 INFO:tasks.rados.rados.0.smithi071.stdout:252: finishing write tid 3 to smithi07183189-10 2023-12-15T22:57:20.820 INFO:tasks.rados.rados.0.smithi071.stdout:250: done (3 left) 2023-12-15T22:57:20.820 INFO:tasks.rados.rados.0.smithi071.stdout:251: done (2 left) 2023-12-15T22:57:20.820 INFO:tasks.rados.rados.0.smithi071.stdout:253: done (1 left) 2023-12-15T22:57:20.821 INFO:tasks.rados.rados.0.smithi071.stdout:254: snap_remove snap 19 2023-12-15T22:57:21.549 INFO:tasks.rados.rados.0.smithi071.stdout:254: done (1 left) 2023-12-15T22:57:21.549 INFO:tasks.rados.rados.0.smithi071.stdout:255: delete oid 39 current snap is 20 2023-12-15T22:57:21.549 INFO:tasks.rados.rados.0.smithi071.stdout:252: finishing write tid 4 to smithi07183189-10 2023-12-15T22:57:21.549 INFO:tasks.rados.rados.0.smithi071.stdout:252: oid 10 updating version 0 to 55 2023-12-15T22:57:21.549 INFO:tasks.rados.rados.0.smithi071.stdout:252: oid 10 version 55 is already newer than 54 2023-12-15T22:57:21.549 INFO:tasks.rados.rados.0.smithi071.stdout:252: oid 10 updating version 55 to 56 2023-12-15T22:57:21.549 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 56 (ObjNum 147 snap 20 seq_num 147) dirty exists 2023-12-15T22:57:21.549 INFO:tasks.rados.rados.0.smithi071.stdout:252: left oid 10 (ObjNum 147 snap 20 seq_num 147) 2023-12-15T22:57:21.579 INFO:tasks.rados.rados.0.smithi071.stdout:252: done (1 left) 2023-12-15T22:57:21.579 INFO:tasks.rados.rados.0.smithi071.stdout:255: done (0 left) 2023-12-15T22:57:21.579 INFO:tasks.rados.rados.0.smithi071.stdout:256: read oid 25 snap 18 2023-12-15T22:57:21.579 INFO:tasks.rados.rados.0.smithi071.stdout:256: expect (ObjNum 137 snap 17 seq_num 137) 2023-12-15T22:57:21.580 INFO:tasks.rados.rados.0.smithi071.stdout:257: write (excl) oid 30 current snap is 20 2023-12-15T22:57:21.580 INFO:tasks.rados.rados.0.smithi071.stdout:257: seq_num 150 ranges {521050=698057,1877946=643104} 2023-12-15T22:57:21.586 INFO:tasks.rados.rados.0.smithi071.stdout:257: writing smithi07183189-30 from 521050 to 1219107 tid 1 2023-12-15T22:57:21.593 INFO:tasks.rados.rados.0.smithi071.stdout:257: writing smithi07183189-30 from 1877946 to 2521050 tid 2 2023-12-15T22:57:21.594 INFO:tasks.rados.rados.0.smithi071.stdout:258: write oid 3 current snap is 20 2023-12-15T22:57:21.594 INFO:tasks.rados.rados.0.smithi071.stdout:258: seq_num 151 ranges {537857=773306,1936428=520715} 2023-12-15T22:57:21.601 INFO:tasks.rados.rados.0.smithi071.stdout:258: writing smithi07183189-3 from 537857 to 1311163 tid 1 2023-12-15T22:57:21.607 INFO:tasks.rados.rados.0.smithi071.stdout:258: writing smithi07183189-3 from 1936428 to 2457143 tid 2 2023-12-15T22:57:21.607 INFO:tasks.rados.rados.0.smithi071.stdout:259: snap_create 2023-12-15T22:57:21.671 INFO:tasks.rados.rados.0.smithi071.stdout:256: expect (ObjNum 137 snap 17 seq_num 137) 2023-12-15T22:57:21.772 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:57:21.817 INFO:tasks.rados.rados.0.smithi071.stdout:257: finishing write tid 1 to smithi07183189-30 2023-12-15T22:57:21.817 INFO:tasks.rados.rados.0.smithi071.stdout:258: finishing write tid 1 to smithi07183189-3 2023-12-15T22:57:21.817 INFO:tasks.rados.rados.0.smithi071.stdout:257: finishing write tid 2 to smithi07183189-30 2023-12-15T22:57:21.817 INFO:tasks.rados.rados.0.smithi071.stdout:258: finishing write tid 2 to smithi07183189-3 2023-12-15T22:57:21.818 INFO:tasks.rados.rados.0.smithi071.stdout:257: finishing write tid 3 to smithi07183189-30 2023-12-15T22:57:21.818 INFO:tasks.rados.rados.0.smithi071.stdout:257: finishing write tid 4 to smithi07183189-30 2023-12-15T22:57:21.818 INFO:tasks.rados.rados.0.smithi071.stdout:257: oid 30 updating version 0 to 25 2023-12-15T22:57:21.818 INFO:tasks.rados.rados.0.smithi071.stdout:257: oid 30 updating version 25 to 26 2023-12-15T22:57:21.818 INFO:tasks.rados.rados.0.smithi071.stdout:257: oid 30 version 26 is already newer than 24 2023-12-15T22:57:21.818 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 26 (ObjNum 150 snap 20 seq_num 150) dirty exists 2023-12-15T22:57:21.818 INFO:tasks.rados.rados.0.smithi071.stdout:257: left oid 30 (ObjNum 150 snap 20 seq_num 150) 2023-12-15T22:57:21.818 INFO:tasks.rados.rados.0.smithi071.stdout:258: finishing write tid 3 to smithi07183189-3 2023-12-15T22:57:21.818 INFO:tasks.rados.rados.0.smithi071.stdout:258: finishing write tid 4 to smithi07183189-3 2023-12-15T22:57:21.818 INFO:tasks.rados.rados.0.smithi071.stdout:258: oid 3 updating version 0 to 31 2023-12-15T22:57:21.819 INFO:tasks.rados.rados.0.smithi071.stdout:258: oid 3 updating version 31 to 33 2023-12-15T22:57:21.819 INFO:tasks.rados.rados.0.smithi071.stdout:258: oid 3 version 33 is already newer than 32 2023-12-15T22:57:21.819 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 33 (ObjNum 151 snap 20 seq_num 151) dirty exists 2023-12-15T22:57:21.819 INFO:tasks.rados.rados.0.smithi071.stdout:258: left oid 3 (ObjNum 151 snap 20 seq_num 151) 2023-12-15T22:57:21.876 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:21.874+0000 7f92b9b1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:21.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:21.879+0000 7f92b9b1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:22.552 INFO:tasks.rados.rados.0.smithi071.stdout:256: done (3 left) 2023-12-15T22:57:22.552 INFO:tasks.rados.rados.0.smithi071.stdout:257: done (2 left) 2023-12-15T22:57:22.552 INFO:tasks.rados.rados.0.smithi071.stdout:258: done (1 left) 2023-12-15T22:57:22.552 INFO:tasks.rados.rados.0.smithi071.stdout:259: done (0 left) 2023-12-15T22:57:22.552 INFO:tasks.rados.rados.0.smithi071.stdout:260: write (excl) oid 45 current snap is 21 2023-12-15T22:57:22.552 INFO:tasks.rados.rados.0.smithi071.stdout:260: seq_num 152 ranges {554664=448555,1478557=664373} 2023-12-15T22:57:22.557 INFO:tasks.rados.rados.0.smithi071.stdout:260: writing smithi07183189-45 from 554664 to 1003219 tid 1 2023-12-15T22:57:22.564 INFO:tasks.rados.rados.0.smithi071.stdout:260: writing smithi07183189-45 from 1478557 to 2142930 tid 2 2023-12-15T22:57:22.564 INFO:tasks.rados.rados.0.smithi071.stdout:261: snap_remove snap 17 2023-12-15T22:57:23.565 INFO:tasks.rados.rados.0.smithi071.stdout:261: done (1 left) 2023-12-15T22:57:23.565 INFO:tasks.rados.rados.0.smithi071.stdout:262: write (excl) oid 9 current snap is 21 2023-12-15T22:57:23.565 INFO:tasks.rados.rados.0.smithi071.stdout:260: finishing write tid 1 to smithi07183189-45 2023-12-15T22:57:23.566 INFO:tasks.rados.rados.0.smithi071.stdout:262: seq_num 153 ranges {571471=640157,1653392=524384} 2023-12-15T22:57:23.571 INFO:tasks.rados.rados.0.smithi071.stdout:262: writing smithi07183189-9 from 571471 to 1211628 tid 1 2023-12-15T22:57:23.577 INFO:tasks.rados.rados.0.smithi071.stdout:262: writing smithi07183189-9 from 1653392 to 2177776 tid 2 2023-12-15T22:57:23.577 INFO:tasks.rados.rados.0.smithi071.stdout:263: write (excl) oid 20 current snap is 21 2023-12-15T22:57:23.577 INFO:tasks.rados.rados.0.smithi071.stdout:260: finishing write tid 2 to smithi07183189-45 2023-12-15T22:57:23.577 INFO:tasks.rados.rados.0.smithi071.stdout:263: seq_num 154 ranges {588278=715406,1711874=668042} 2023-12-15T22:57:23.584 INFO:tasks.rados.rados.0.smithi071.stdout:263: writing smithi07183189-20 from 588278 to 1303684 tid 1 2023-12-15T22:57:23.591 INFO:tasks.rados.rados.0.smithi071.stdout:263: writing smithi07183189-20 from 1711874 to 2379916 tid 2 2023-12-15T22:57:23.591 INFO:tasks.rados.rados.0.smithi071.stdout:264: snap_create 2023-12-15T22:57:23.591 INFO:tasks.rados.rados.0.smithi071.stdout:260: finishing write tid 3 to smithi07183189-45 2023-12-15T22:57:23.591 INFO:tasks.rados.rados.0.smithi071.stdout:260: finishing write tid 4 to smithi07183189-45 2023-12-15T22:57:23.591 INFO:tasks.rados.rados.0.smithi071.stdout:260: oid 45 updating version 0 to 129 2023-12-15T22:57:23.591 INFO:tasks.rados.rados.0.smithi071.stdout:260: oid 45 version 129 is already newer than 127 2023-12-15T22:57:23.591 INFO:tasks.rados.rados.0.smithi071.stdout:260: oid 45 version 129 is already newer than 129 2023-12-15T22:57:23.591 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 129 (ObjNum 152 snap 21 seq_num 152) dirty exists 2023-12-15T22:57:23.591 INFO:tasks.rados.rados.0.smithi071.stdout:260: left oid 45 (ObjNum 152 snap 21 seq_num 152) 2023-12-15T22:57:23.656 INFO:tasks.rados.rados.0.smithi071.stdout:262: finishing write tid 1 to smithi07183189-9 2023-12-15T22:57:23.667 INFO:tasks.rados.rados.0.smithi071.stdout:263: finishing write tid 1 to smithi07183189-20 2023-12-15T22:57:23.687 INFO:tasks.rados.rados.0.smithi071.stdout:262: finishing write tid 2 to smithi07183189-9 2023-12-15T22:57:23.688 INFO:tasks.rados.rados.0.smithi071.stdout:262: finishing write tid 3 to smithi07183189-9 2023-12-15T22:57:23.690 INFO:tasks.rados.rados.0.smithi071.stdout:263: finishing write tid 2 to smithi07183189-20 2023-12-15T22:57:23.690 INFO:tasks.rados.rados.0.smithi071.stdout:262: finishing write tid 4 to smithi07183189-9 2023-12-15T22:57:23.690 INFO:tasks.rados.rados.0.smithi071.stdout:262: oid 9 updating version 0 to 39 2023-12-15T22:57:23.690 INFO:tasks.rados.rados.0.smithi071.stdout:262: oid 9 updating version 39 to 41 2023-12-15T22:57:23.690 INFO:tasks.rados.rados.0.smithi071.stdout:262: oid 9 version 41 is already newer than 41 2023-12-15T22:57:23.690 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 41 (ObjNum 153 snap 21 seq_num 153) dirty exists 2023-12-15T22:57:23.690 INFO:tasks.rados.rados.0.smithi071.stdout:262: left oid 9 (ObjNum 153 snap 21 seq_num 153) 2023-12-15T22:57:23.697 INFO:tasks.rados.rados.0.smithi071.stdout:263: finishing write tid 3 to smithi07183189-20 2023-12-15T22:57:23.698 INFO:tasks.rados.rados.0.smithi071.stdout:263: finishing write tid 4 to smithi07183189-20 2023-12-15T22:57:23.698 INFO:tasks.rados.rados.0.smithi071.stdout:263: oid 20 updating version 0 to 136 2023-12-15T22:57:23.698 INFO:tasks.rados.rados.0.smithi071.stdout:263: oid 20 version 136 is already newer than 135 2023-12-15T22:57:23.698 INFO:tasks.rados.rados.0.smithi071.stdout:263: oid 20 updating version 136 to 137 2023-12-15T22:57:23.698 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 20 v 137 (ObjNum 154 snap 21 seq_num 154) dirty exists 2023-12-15T22:57:23.698 INFO:tasks.rados.rados.0.smithi071.stdout:263: left oid 20 (ObjNum 154 snap 21 seq_num 154) 2023-12-15T22:57:23.948 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:57:24.052 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:24.051+0000 7feae0755640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:24.055 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:24.055+0000 7feae0755640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:24.574 INFO:tasks.rados.rados.0.smithi071.stdout:260: done (3 left) 2023-12-15T22:57:24.574 INFO:tasks.rados.rados.0.smithi071.stdout:262: done (2 left) 2023-12-15T22:57:24.574 INFO:tasks.rados.rados.0.smithi071.stdout:263: done (1 left) 2023-12-15T22:57:24.575 INFO:tasks.rados.rados.0.smithi071.stdout:264: done (0 left) 2023-12-15T22:57:24.575 INFO:tasks.rados.rados.0.smithi071.stdout:265: snap_create 2023-12-15T22:57:25.253 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T22:57:25.253 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T22:57:25.253 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-12-15T22:57:25.254 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-12-15T22:57:25.373 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:25.368+0000 7f906b4de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:25.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:25.376+0000 7f906b4de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:25.388 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:57:25.388 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:57:25.388 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:57:25.388 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:57:25.388 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:57:25.389 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:57:25.389 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:57:25.389 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:57:25.389 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:57:25.389 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:57:25.389 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:57:25.389 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:57:25.392 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:57:25.392 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:57:25.392 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-12-15T22:57:25.499 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:25.494+0000 7f3586f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:25.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:25.502+0000 7f3586f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:25.514 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:57:25.514 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:57:25.514 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:57:25.514 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:57:25.515 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:57:25.515 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:57:25.515 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:57:25.515 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:57:25.515 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:57:25.515 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:57:25.515 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:57:25.515 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:57:25.517 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:57:25.518 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:57:25.518 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-12-15T22:57:25.573 INFO:tasks.rados.rados.0.smithi071.stdout:265: done (0 left) 2023-12-15T22:57:25.573 INFO:tasks.rados.rados.0.smithi071.stdout:266: snap_remove snap 22 2023-12-15T22:57:25.628 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:25.623+0000 7fb6dc66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:25.631 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:25.631+0000 7fb6dc66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:25.644 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:57:25.644 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:57:25.644 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:57:25.644 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:57:25.644 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:57:25.644 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:57:25.644 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:57:25.644 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:57:25.644 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:57:25.645 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:57:25.645 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:57:25.645 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:57:25.647 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:57:25.647 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:57:25.647 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-12-15T22:57:25.753 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:25.749+0000 7fe44866d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:25.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:25.756+0000 7fe44866d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:25.769 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:57:25.769 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:57:25.769 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:57:25.769 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:57:25.770 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:57:25.770 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:57:25.770 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:57:25.770 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:57:25.770 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:57:25.770 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:57:25.770 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:57:25.770 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:57:25.772 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:57:25.772 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:57:26.126 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:57:26.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:26.228+0000 7fd6d3f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:26.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:26.232+0000 7fd6d3f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:26.571 INFO:tasks.rados.rados.0.smithi071.stdout:266: done (0 left) 2023-12-15T22:57:26.572 INFO:tasks.rados.rados.0.smithi071.stdout:267: write (excl) oid 9 current snap is 23 2023-12-15T22:57:26.572 INFO:tasks.rados.rados.0.smithi071.stdout:267: seq_num 155 ranges {605085=790655,2170356=434729} 2023-12-15T22:57:26.581 INFO:tasks.rados.rados.0.smithi071.stdout:267: writing smithi07183189-9 from 605085 to 1395740 tid 1 2023-12-15T22:57:26.588 INFO:tasks.rados.rados.0.smithi071.stdout:267: writing smithi07183189-9 from 2170356 to 2605085 tid 2 2023-12-15T22:57:26.588 INFO:tasks.rados.rados.0.smithi071.stdout:268: snap_create 2023-12-15T22:57:26.631 INFO:tasks.rados.rados.0.smithi071.stdout:267: finishing write tid 1 to smithi07183189-9 2023-12-15T22:57:26.648 INFO:tasks.rados.rados.0.smithi071.stdout:267: finishing write tid 2 to smithi07183189-9 2023-12-15T22:57:26.657 INFO:tasks.rados.rados.0.smithi071.stdout:267: finishing write tid 3 to smithi07183189-9 2023-12-15T22:57:26.658 INFO:tasks.rados.rados.0.smithi071.stdout:267: finishing write tid 4 to smithi07183189-9 2023-12-15T22:57:26.658 INFO:tasks.rados.rados.0.smithi071.stdout:267: oid 9 updating version 0 to 44 2023-12-15T22:57:26.658 INFO:tasks.rados.rados.0.smithi071.stdout:267: oid 9 updating version 44 to 45 2023-12-15T22:57:26.659 INFO:tasks.rados.rados.0.smithi071.stdout:267: oid 9 version 45 is already newer than 43 2023-12-15T22:57:26.659 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 45 (ObjNum 155 snap 23 seq_num 155) dirty exists 2023-12-15T22:57:26.659 INFO:tasks.rados.rados.0.smithi071.stdout:267: left oid 9 (ObjNum 155 snap 23 seq_num 155) 2023-12-15T22:57:27.583 INFO:tasks.rados.rados.0.smithi071.stdout:267: done (1 left) 2023-12-15T22:57:27.583 INFO:tasks.rados.rados.0.smithi071.stdout:268: done (0 left) 2023-12-15T22:57:27.584 INFO:tasks.rados.rados.0.smithi071.stdout:269: read oid 33 snap -1 2023-12-15T22:57:27.584 INFO:tasks.rados.rados.0.smithi071.stdout:269: expect (ObjNum 124 snap 15 seq_num 124) 2023-12-15T22:57:27.584 INFO:tasks.rados.rados.0.smithi071.stdout:270: delete oid 39 current snap is 24 2023-12-15T22:57:27.614 INFO:tasks.rados.rados.0.smithi071.stdout:269: expect (ObjNum 124 snap 15 seq_num 124) 2023-12-15T22:57:27.726 INFO:tasks.rados.rados.0.smithi071.stdout:269: done (1 left) 2023-12-15T22:57:27.726 INFO:tasks.rados.rados.0.smithi071.stdout:270: done (0 left) 2023-12-15T22:57:27.726 INFO:tasks.rados.rados.0.smithi071.stdout:271: snap_remove snap 21 2023-12-15T22:57:28.301 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:57:28.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:28.405+0000 7f7beba2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:28.409 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:28.409+0000 7f7beba2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:28.585 INFO:tasks.rados.rados.0.smithi071.stdout:271: done (0 left) 2023-12-15T22:57:28.585 INFO:tasks.rados.rados.0.smithi071.stdout:272: write (excl) oid 2 current snap is 24 2023-12-15T22:57:28.585 INFO:tasks.rados.rados.0.smithi071.stdout:272: seq_num 157 ranges {638699=541153,1770967=531722} 2023-12-15T22:57:28.591 INFO:tasks.rados.rados.0.smithi071.stdout:272: writing smithi07183189-2 from 638699 to 1179852 tid 1 2023-12-15T22:57:28.597 INFO:tasks.rados.rados.0.smithi071.stdout:272: writing smithi07183189-2 from 1770967 to 2302689 tid 2 2023-12-15T22:57:28.597 INFO:tasks.rados.rados.0.smithi071.stdout:273: delete oid 8 current snap is 24 2023-12-15T22:57:28.656 INFO:tasks.rados.rados.0.smithi071.stdout:273: done (1 left) 2023-12-15T22:57:28.656 INFO:tasks.rados.rados.0.smithi071.stdout:274: delete oid 42 current snap is 24 2023-12-15T22:57:28.667 INFO:tasks.rados.rados.0.smithi071.stdout:272: finishing write tid 1 to smithi07183189-2 2023-12-15T22:57:28.702 INFO:tasks.rados.rados.0.smithi071.stdout:274: done (1 left) 2023-12-15T22:57:28.702 INFO:tasks.rados.rados.0.smithi071.stdout:275: read oid 27 snap -1 2023-12-15T22:57:28.702 INFO:tasks.rados.rados.0.smithi071.stdout:275: expect (ObjNum 74 snap 3 seq_num 74) 2023-12-15T22:57:28.702 INFO:tasks.rados.rados.0.smithi071.stdout:276: snap_remove snap 20 2023-12-15T22:57:29.587 INFO:tasks.rados.rados.0.smithi071.stdout:276: done (2 left) 2023-12-15T22:57:29.588 INFO:tasks.rados.rados.0.smithi071.stdout:277: delete oid 32 current snap is 24 2023-12-15T22:57:29.588 INFO:tasks.rados.rados.0.smithi071.stdout:272: finishing write tid 2 to smithi07183189-2 2023-12-15T22:57:29.588 INFO:tasks.rados.rados.0.smithi071.stdout:272: finishing write tid 3 to smithi07183189-2 2023-12-15T22:57:29.588 INFO:tasks.rados.rados.0.smithi071.stdout:272: finishing write tid 4 to smithi07183189-2 2023-12-15T22:57:29.588 INFO:tasks.rados.rados.0.smithi071.stdout:272: oid 2 updating version 0 to 28 2023-12-15T22:57:29.588 INFO:tasks.rados.rados.0.smithi071.stdout:272: oid 2 updating version 28 to 29 2023-12-15T22:57:29.588 INFO:tasks.rados.rados.0.smithi071.stdout:272: oid 2 updating version 29 to 30 2023-12-15T22:57:29.588 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 30 (ObjNum 157 snap 24 seq_num 157) dirty exists 2023-12-15T22:57:29.588 INFO:tasks.rados.rados.0.smithi071.stdout:272: left oid 2 (ObjNum 157 snap 24 seq_num 157) 2023-12-15T22:57:29.588 INFO:tasks.rados.rados.0.smithi071.stdout:275: expect (ObjNum 74 snap 3 seq_num 74) 2023-12-15T22:57:29.775 INFO:tasks.rados.rados.0.smithi071.stdout:272: done (2 left) 2023-12-15T22:57:29.776 INFO:tasks.rados.rados.0.smithi071.stdout:275: done (1 left) 2023-12-15T22:57:29.776 INFO:tasks.rados.rados.0.smithi071.stdout:277: done (0 left) 2023-12-15T22:57:29.776 INFO:tasks.rados.rados.0.smithi071.stdout:278: write (excl) oid 37 current snap is 24 2023-12-15T22:57:29.776 INFO:tasks.rados.rados.0.smithi071.stdout:278: seq_num 161 ranges {705927=558502,2004895=422707} 2023-12-15T22:57:29.782 INFO:tasks.rados.rados.0.smithi071.stdout:278: writing smithi07183189-37 from 705927 to 1264429 tid 1 2023-12-15T22:57:29.789 INFO:tasks.rados.rados.0.smithi071.stdout:278: writing smithi07183189-37 from 2004895 to 2427602 tid 2 2023-12-15T22:57:29.789 INFO:tasks.rados.rados.0.smithi071.stdout:279: write (excl) oid 50 current snap is 24 2023-12-15T22:57:29.789 INFO:tasks.rados.rados.0.smithi071.stdout:279: seq_num 162 ranges {722734=633751,2063377=659357} 2023-12-15T22:57:29.796 INFO:tasks.rados.rados.0.smithi071.stdout:279: writing smithi07183189-50 from 722734 to 1356485 tid 1 2023-12-15T22:57:29.804 INFO:tasks.rados.rados.0.smithi071.stdout:279: writing smithi07183189-50 from 2063377 to 2722734 tid 2 2023-12-15T22:57:29.804 INFO:tasks.rados.rados.0.smithi071.stdout:280: snap_remove snap 18 2023-12-15T22:57:30.479 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:57:30.585 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:30.583+0000 7f3ea826d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:30.588 INFO:tasks.rados.rados.0.smithi071.stdout:278: finishing write tid 1 to smithi07183189-37 2023-12-15T22:57:30.588 INFO:tasks.rados.rados.0.smithi071.stdout:278: finishing write tid 2 to smithi07183189-37 2023-12-15T22:57:30.589 INFO:tasks.rados.rados.0.smithi071.stdout:278: finishing write tid 3 to smithi07183189-37 2023-12-15T22:57:30.589 INFO:tasks.rados.rados.0.smithi071.stdout:278: finishing write tid 4 to smithi07183189-37 2023-12-15T22:57:30.589 INFO:tasks.rados.rados.0.smithi071.stdout:278: oid 37 updating version 0 to 143 2023-12-15T22:57:30.589 INFO:tasks.rados.rados.0.smithi071.stdout:278: oid 37 updating version 143 to 145 2023-12-15T22:57:30.589 INFO:tasks.rados.rados.0.smithi071.stdout:278: oid 37 version 145 is already newer than 144 2023-12-15T22:57:30.589 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 145 (ObjNum 161 snap 24 seq_num 161) dirty exists 2023-12-15T22:57:30.589 INFO:tasks.rados.rados.0.smithi071.stdout:278: left oid 37 (ObjNum 161 snap 24 seq_num 161) 2023-12-15T22:57:30.589 INFO:tasks.rados.rados.0.smithi071.stdout:279: finishing write tid 1 to smithi07183189-50 2023-12-15T22:57:30.589 INFO:tasks.rados.rados.0.smithi071.stdout:278: done (2 left) 2023-12-15T22:57:30.589 INFO:tasks.rados.rados.0.smithi071.stdout:280: done (1 left) 2023-12-15T22:57:30.589 INFO:tasks.rados.rados.0.smithi071.stdout:281: write (excl) oid 47 current snap is 24 2023-12-15T22:57:30.589 INFO:tasks.rados.rados.0.smithi071.stdout:279: finishing write tid 2 to smithi07183189-50 2023-12-15T22:57:30.589 INFO:tasks.rados.rados.0.smithi071.stdout:279: finishing write tid 3 to smithi07183189-50 2023-12-15T22:57:30.589 INFO:tasks.rados.rados.0.smithi071.stdout:281: seq_num 163 ranges {739541=709000,2121859=426376} 2023-12-15T22:57:30.590 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:30.589+0000 7f3ea826d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:30.597 INFO:tasks.rados.rados.0.smithi071.stdout:281: writing smithi07183189-47 from 739541 to 1448541 tid 1 2023-12-15T22:57:30.603 INFO:tasks.rados.rados.0.smithi071.stdout:281: writing smithi07183189-47 from 2121859 to 2548235 tid 2 2023-12-15T22:57:30.603 INFO:tasks.rados.rados.0.smithi071.stdout:282: write oid 34 current snap is 24 2023-12-15T22:57:30.604 INFO:tasks.rados.rados.0.smithi071.stdout:279: finishing write tid 4 to smithi07183189-50 2023-12-15T22:57:30.604 INFO:tasks.rados.rados.0.smithi071.stdout:279: oid 50 updating version 0 to 148 2023-12-15T22:57:30.604 INFO:tasks.rados.rados.0.smithi071.stdout:279: oid 50 version 148 is already newer than 148 2023-12-15T22:57:30.604 INFO:tasks.rados.rados.0.smithi071.stdout:279: oid 50 version 148 is already newer than 146 2023-12-15T22:57:30.604 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 148 (ObjNum 162 snap 24 seq_num 162) dirty exists 2023-12-15T22:57:30.604 INFO:tasks.rados.rados.0.smithi071.stdout:279: left oid 50 (ObjNum 162 snap 24 seq_num 162) 2023-12-15T22:57:30.604 INFO:tasks.rados.rados.0.smithi071.stdout:282: seq_num 164 ranges {756348=784249,2063988=686387} 2023-12-15T22:57:30.612 INFO:tasks.rados.rados.0.smithi071.stdout:282: writing smithi07183189-34 from 756348 to 1540597 tid 1 2023-12-15T22:57:30.619 INFO:tasks.rados.rados.0.smithi071.stdout:282: writing smithi07183189-34 from 2063988 to 2750375 tid 2 2023-12-15T22:57:30.619 INFO:tasks.rados.rados.0.smithi071.stdout:279: done (2 left) 2023-12-15T22:57:30.619 INFO:tasks.rados.rados.0.smithi071.stdout:283: snap_create 2023-12-15T22:57:30.698 INFO:tasks.rados.rados.0.smithi071.stdout:281: finishing write tid 1 to smithi07183189-47 2023-12-15T22:57:30.698 INFO:tasks.rados.rados.0.smithi071.stdout:282: finishing write tid 1 to smithi07183189-34 2023-12-15T22:57:30.703 INFO:tasks.rados.rados.0.smithi071.stdout:282: finishing write tid 2 to smithi07183189-34 2023-12-15T22:57:30.720 INFO:tasks.rados.rados.0.smithi071.stdout:282: finishing write tid 3 to smithi07183189-34 2023-12-15T22:57:30.728 INFO:tasks.rados.rados.0.smithi071.stdout:282: finishing write tid 4 to smithi07183189-34 2023-12-15T22:57:30.728 INFO:tasks.rados.rados.0.smithi071.stdout:282: oid 34 updating version 0 to 6 2023-12-15T22:57:30.728 INFO:tasks.rados.rados.0.smithi071.stdout:282: oid 34 version 6 is already newer than 6 2023-12-15T22:57:30.729 INFO:tasks.rados.rados.0.smithi071.stdout:282: oid 34 updating version 6 to 7 2023-12-15T22:57:30.729 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 7 (ObjNum 164 snap 24 seq_num 164) dirty exists 2023-12-15T22:57:30.729 INFO:tasks.rados.rados.0.smithi071.stdout:282: left oid 34 (ObjNum 164 snap 24 seq_num 164) 2023-12-15T22:57:30.730 INFO:tasks.rados.rados.0.smithi071.stdout:281: finishing write tid 2 to smithi07183189-47 2023-12-15T22:57:30.738 INFO:tasks.rados.rados.0.smithi071.stdout:281: finishing write tid 3 to smithi07183189-47 2023-12-15T22:57:30.739 INFO:tasks.rados.rados.0.smithi071.stdout:281: finishing write tid 4 to smithi07183189-47 2023-12-15T22:57:30.739 INFO:tasks.rados.rados.0.smithi071.stdout:281: oid 47 updating version 0 to 23 2023-12-15T22:57:30.739 INFO:tasks.rados.rados.0.smithi071.stdout:281: oid 47 version 23 is already newer than 21 2023-12-15T22:57:30.739 INFO:tasks.rados.rados.0.smithi071.stdout:281: oid 47 version 23 is already newer than 22 2023-12-15T22:57:30.739 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 23 (ObjNum 163 snap 24 seq_num 163) dirty exists 2023-12-15T22:57:30.739 INFO:tasks.rados.rados.0.smithi071.stdout:281: left oid 47 (ObjNum 163 snap 24 seq_num 163) 2023-12-15T22:57:30.773 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T22:57:30.773 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2023-12-15T22:57:30.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:30.873+0000 7f381ad5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:30.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:30.881+0000 7f381ad5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:31.587 INFO:teuthology.orchestra.run.smithi071.stderr:reweighted osd.0 to 1 (10000) 2023-12-15T22:57:31.590 INFO:tasks.rados.rados.0.smithi071.stdout:281: done (2 left) 2023-12-15T22:57:31.590 INFO:tasks.rados.rados.0.smithi071.stdout:282: done (1 left) 2023-12-15T22:57:31.590 INFO:tasks.rados.rados.0.smithi071.stdout:283: done (0 left) 2023-12-15T22:57:31.590 INFO:tasks.rados.rados.0.smithi071.stdout:284: read oid 13 snap -1 2023-12-15T22:57:31.590 INFO:tasks.rados.rados.0.smithi071.stdout:284: expect deleted 2023-12-15T22:57:31.590 INFO:tasks.rados.rados.0.smithi071.stdout:285: snap_remove snap 24 2023-12-15T22:57:31.601 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2023-12-15T22:57:31.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:31.702+0000 7f8edf91e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:31.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:31.711+0000 7f8edf91e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:32.595 INFO:teuthology.orchestra.run.smithi071.stderr:reweighted osd.1 to 1 (10000) 2023-12-15T22:57:32.598 INFO:tasks.rados.rados.0.smithi071.stdout:285: done (1 left) 2023-12-15T22:57:32.598 INFO:tasks.rados.rados.0.smithi071.stdout:286: write oid 9 current snap is 25 2023-12-15T22:57:32.598 INFO:tasks.rados.rados.0.smithi071.stdout:286: seq_num 165 ranges {773155=459498,1722470=430045,2648676=124479} 2023-12-15T22:57:32.604 INFO:tasks.rados.rados.0.smithi071.stdout:286: writing smithi07183189-9 from 773155 to 1232653 tid 1 2023-12-15T22:57:32.609 INFO:tasks.rados.rados.0.smithi071.stdout:286: writing smithi07183189-9 from 1722470 to 2152515 tid 2 2023-12-15T22:57:32.612 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2023-12-15T22:57:32.614 INFO:tasks.rados.rados.0.smithi071.stdout:286: writing smithi07183189-9 from 2648676 to 2773155 tid 3 2023-12-15T22:57:32.614 INFO:tasks.rados.rados.0.smithi071.stdout:287: delete oid 26 current snap is 25 2023-12-15T22:57:32.707 INFO:tasks.rados.rados.0.smithi071.stdout:284: done (2 left) 2023-12-15T22:57:32.707 INFO:tasks.rados.rados.0.smithi071.stdout:287: done (1 left) 2023-12-15T22:57:32.707 INFO:tasks.rados.rados.0.smithi071.stdout:288: read oid 6 snap 23 2023-12-15T22:57:32.707 INFO:tasks.rados.rados.0.smithi071.stdout:288: expect (ObjNum 66 snap 3 seq_num 66) 2023-12-15T22:57:32.707 INFO:tasks.rados.rados.0.smithi071.stdout:289: snap_create 2023-12-15T22:57:32.716 INFO:tasks.rados.rados.0.smithi071.stdout:286: finishing write tid 1 to smithi07183189-9 2023-12-15T22:57:32.733 INFO:tasks.rados.rados.0.smithi071.stdout:286: finishing write tid 2 to smithi07183189-9 2023-12-15T22:57:32.736 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:32.729+0000 7f040d260640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:32.739 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:57:32.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:32.738+0000 7f040d260640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:32.767 INFO:tasks.rados.rados.0.smithi071.stdout:286: finishing write tid 3 to smithi07183189-9 2023-12-15T22:57:32.772 INFO:tasks.rados.rados.0.smithi071.stdout:288: expect (ObjNum 66 snap 3 seq_num 66) 2023-12-15T22:57:32.857 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:32.854+0000 7f318779e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:32.860 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:32.859+0000 7f318779e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:32.949 INFO:tasks.rados.rados.0.smithi071.stdout:286: finishing write tid 4 to smithi07183189-9 2023-12-15T22:57:32.949 INFO:tasks.rados.rados.0.smithi071.stdout:286: finishing write tid 5 to smithi07183189-9 2023-12-15T22:57:32.949 INFO:tasks.rados.rados.0.smithi071.stdout:286: oid 9 updating version 0 to 51 2023-12-15T22:57:32.949 INFO:tasks.rados.rados.0.smithi071.stdout:286: oid 9 updating version 51 to 55 2023-12-15T22:57:32.949 INFO:tasks.rados.rados.0.smithi071.stdout:286: oid 9 version 55 is already newer than 52 2023-12-15T22:57:32.949 INFO:tasks.rados.rados.0.smithi071.stdout:286: oid 9 updating version 55 to 56 2023-12-15T22:57:32.949 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 56 (ObjNum 165 snap 25 seq_num 165) dirty exists 2023-12-15T22:57:32.949 INFO:tasks.rados.rados.0.smithi071.stdout:286: left oid 9 (ObjNum 165 snap 25 seq_num 165) 2023-12-15T22:57:33.597 INFO:teuthology.orchestra.run.smithi071.stderr:reweighted osd.2 to 1 (10000) 2023-12-15T22:57:33.600 INFO:tasks.rados.rados.0.smithi071.stdout:286: done (2 left) 2023-12-15T22:57:33.600 INFO:tasks.rados.rados.0.smithi071.stdout:288: done (1 left) 2023-12-15T22:57:33.600 INFO:tasks.rados.rados.0.smithi071.stdout:289: done (0 left) 2023-12-15T22:57:33.600 INFO:tasks.rados.rados.0.smithi071.stdout:290: write (excl) oid 7 current snap is 26 2023-12-15T22:57:33.600 INFO:tasks.rados.rados.0.smithi071.stdout:290: seq_num 167 ranges {406769=609996,1439434=433714,2303522=503247} 2023-12-15T22:57:33.606 INFO:tasks.rados.rados.0.smithi071.stdout:290: writing smithi07183189-7 from 406769 to 1016765 tid 1 2023-12-15T22:57:33.611 INFO:tasks.rados.rados.0.smithi071.stdout:290: writing smithi07183189-7 from 1439434 to 1873148 tid 2 2023-12-15T22:57:33.612 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2023-12-15T22:57:33.617 INFO:tasks.rados.rados.0.smithi071.stdout:290: writing smithi07183189-7 from 2303522 to 2806769 tid 3 2023-12-15T22:57:33.618 INFO:tasks.rados.rados.0.smithi071.stdout:291: read oid 50 snap -1 2023-12-15T22:57:33.618 INFO:tasks.rados.rados.0.smithi071.stdout:291: expect (ObjNum 162 snap 24 seq_num 162) 2023-12-15T22:57:33.618 INFO:tasks.rados.rados.0.smithi071.stdout:292: read oid 36 snap -1 2023-12-15T22:57:33.618 INFO:tasks.rados.rados.0.smithi071.stdout:292: expect deleted 2023-12-15T22:57:33.618 INFO:tasks.rados.rados.0.smithi071.stdout:293: read oid 20 snap 23 2023-12-15T22:57:33.618 INFO:tasks.rados.rados.0.smithi071.stdout:293: expect (ObjNum 154 snap 21 seq_num 154) 2023-12-15T22:57:33.618 INFO:tasks.rados.rados.0.smithi071.stdout:294: delete oid 23 current snap is 26 2023-12-15T22:57:33.644 INFO:tasks.rados.rados.0.smithi071.stdout:294: done (4 left) 2023-12-15T22:57:33.644 INFO:tasks.rados.rados.0.smithi071.stdout:295: write oid 3 current snap is 26 2023-12-15T22:57:33.644 INFO:tasks.rados.rados.0.smithi071.stdout:295: seq_num 169 ranges {440383=476847,1556398=437383,2474721=365662} 2023-12-15T22:57:33.649 INFO:tasks.rados.rados.0.smithi071.stdout:295: writing smithi07183189-3 from 440383 to 917230 tid 1 2023-12-15T22:57:33.656 INFO:tasks.rados.rados.0.smithi071.stdout:295: writing smithi07183189-3 from 1556398 to 1993781 tid 2 2023-12-15T22:57:33.661 INFO:tasks.rados.rados.0.smithi071.stdout:295: writing smithi07183189-3 from 2474721 to 2840383 tid 3 2023-12-15T22:57:33.661 INFO:tasks.rados.rados.0.smithi071.stdout:296: write oid 9 current snap is 26 2023-12-15T22:57:33.661 INFO:tasks.rados.rados.0.smithi071.stdout:296: seq_num 170 ranges {457190=552096,1614880=581041,2785791=71399} 2023-12-15T22:57:33.666 INFO:tasks.rados.rados.0.smithi071.stdout:296: writing smithi07183189-9 from 457190 to 1009286 tid 1 2023-12-15T22:57:33.673 INFO:tasks.rados.rados.0.smithi071.stdout:296: writing smithi07183189-9 from 1614880 to 2195921 tid 2 2023-12-15T22:57:33.677 INFO:tasks.rados.rados.0.smithi071.stdout:296: writing smithi07183189-9 from 2785791 to 2857190 tid 3 2023-12-15T22:57:33.677 INFO:tasks.rados.rados.0.smithi071.stdout:297: snap_create 2023-12-15T22:57:33.677 INFO:tasks.rados.rados.0.smithi071.stdout:291: expect (ObjNum 162 snap 24 seq_num 162) 2023-12-15T22:57:33.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:33.734+0000 7fa7d219e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:33.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:33.743+0000 7fa7d219e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:33.798 INFO:tasks.rados.rados.0.smithi071.stdout:293: expect (ObjNum 154 snap 21 seq_num 154) 2023-12-15T22:57:33.939 INFO:tasks.rados.rados.0.smithi071.stdout:290: finishing write tid 1 to smithi07183189-7 2023-12-15T22:57:33.939 INFO:tasks.rados.rados.0.smithi071.stdout:296: finishing write tid 1 to smithi07183189-9 2023-12-15T22:57:33.939 INFO:tasks.rados.rados.0.smithi071.stdout:295: finishing write tid 1 to smithi07183189-3 2023-12-15T22:57:33.939 INFO:tasks.rados.rados.0.smithi071.stdout:290: finishing write tid 2 to smithi07183189-7 2023-12-15T22:57:33.939 INFO:tasks.rados.rados.0.smithi071.stdout:295: finishing write tid 2 to smithi07183189-3 2023-12-15T22:57:33.939 INFO:tasks.rados.rados.0.smithi071.stdout:296: finishing write tid 2 to smithi07183189-9 2023-12-15T22:57:33.939 INFO:tasks.rados.rados.0.smithi071.stdout:290: finishing write tid 3 to smithi07183189-7 2023-12-15T22:57:33.940 INFO:tasks.rados.rados.0.smithi071.stdout:290: finishing write tid 4 to smithi07183189-7 2023-12-15T22:57:33.940 INFO:tasks.rados.rados.0.smithi071.stdout:290: finishing write tid 5 to smithi07183189-7 2023-12-15T22:57:33.940 INFO:tasks.rados.rados.0.smithi071.stdout:290: oid 7 updating version 0 to 92 2023-12-15T22:57:33.940 INFO:tasks.rados.rados.0.smithi071.stdout:290: oid 7 updating version 92 to 94 2023-12-15T22:57:33.940 INFO:tasks.rados.rados.0.smithi071.stdout:290: oid 7 version 94 is already newer than 94 2023-12-15T22:57:33.940 INFO:tasks.rados.rados.0.smithi071.stdout:290: oid 7 version 94 is already newer than 91 2023-12-15T22:57:33.940 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 94 (ObjNum 167 snap 26 seq_num 167) dirty exists 2023-12-15T22:57:33.940 INFO:tasks.rados.rados.0.smithi071.stdout:290: left oid 7 (ObjNum 167 snap 26 seq_num 167) 2023-12-15T22:57:33.940 INFO:tasks.rados.rados.0.smithi071.stdout:296: finishing write tid 3 to smithi07183189-9 2023-12-15T22:57:33.940 INFO:tasks.rados.rados.0.smithi071.stdout:295: finishing write tid 3 to smithi07183189-3 2023-12-15T22:57:33.940 INFO:tasks.rados.rados.0.smithi071.stdout:296: finishing write tid 4 to smithi07183189-9 2023-12-15T22:57:33.940 INFO:tasks.rados.rados.0.smithi071.stdout:295: finishing write tid 4 to smithi07183189-3 2023-12-15T22:57:33.940 INFO:tasks.rados.rados.0.smithi071.stdout:296: finishing write tid 5 to smithi07183189-9 2023-12-15T22:57:33.940 INFO:tasks.rados.rados.0.smithi071.stdout:296: oid 9 updating version 0 to 59 2023-12-15T22:57:33.940 INFO:tasks.rados.rados.0.smithi071.stdout:296: oid 9 updating version 59 to 60 2023-12-15T22:57:33.941 INFO:tasks.rados.rados.0.smithi071.stdout:296: oid 9 version 60 is already newer than 58 2023-12-15T22:57:33.941 INFO:tasks.rados.rados.0.smithi071.stdout:296: oid 9 updating version 60 to 61 2023-12-15T22:57:33.941 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 61 (ObjNum 170 snap 26 seq_num 170) dirty exists 2023-12-15T22:57:33.941 INFO:tasks.rados.rados.0.smithi071.stdout:296: left oid 9 (ObjNum 170 snap 26 seq_num 170) 2023-12-15T22:57:33.941 INFO:tasks.rados.rados.0.smithi071.stdout:295: finishing write tid 5 to smithi07183189-3 2023-12-15T22:57:33.941 INFO:tasks.rados.rados.0.smithi071.stdout:295: oid 3 updating version 0 to 39 2023-12-15T22:57:33.941 INFO:tasks.rados.rados.0.smithi071.stdout:295: oid 3 updating version 39 to 40 2023-12-15T22:57:33.941 INFO:tasks.rados.rados.0.smithi071.stdout:295: oid 3 updating version 40 to 41 2023-12-15T22:57:33.941 INFO:tasks.rados.rados.0.smithi071.stdout:295: oid 3 updating version 41 to 42 2023-12-15T22:57:33.941 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 42 (ObjNum 169 snap 26 seq_num 169) dirty exists 2023-12-15T22:57:33.941 INFO:tasks.rados.rados.0.smithi071.stdout:295: left oid 3 (ObjNum 169 snap 26 seq_num 169) 2023-12-15T22:57:34.606 INFO:teuthology.orchestra.run.smithi071.stderr:reweighted osd.3 to 1 (10000) 2023-12-15T22:57:34.609 INFO:tasks.rados.rados.0.smithi071.stdout:290: done (6 left) 2023-12-15T22:57:34.610 INFO:tasks.rados.rados.0.smithi071.stdout:291: done (5 left) 2023-12-15T22:57:34.610 INFO:tasks.rados.rados.0.smithi071.stdout:292: done (4 left) 2023-12-15T22:57:34.610 INFO:tasks.rados.rados.0.smithi071.stdout:293: done (3 left) 2023-12-15T22:57:34.610 INFO:tasks.rados.rados.0.smithi071.stdout:295: done (2 left) 2023-12-15T22:57:34.610 INFO:tasks.rados.rados.0.smithi071.stdout:296: done (1 left) 2023-12-15T22:57:34.610 INFO:tasks.rados.rados.0.smithi071.stdout:297: done (0 left) 2023-12-15T22:57:34.610 INFO:tasks.rados.rados.0.smithi071.stdout:298: read oid 24 snap 25 2023-12-15T22:57:34.610 INFO:tasks.rados.rados.0.smithi071.stdout:298: expect deleted 2023-12-15T22:57:34.610 INFO:tasks.rados.rados.0.smithi071.stdout:299: snap_remove snap 23 2023-12-15T22:57:34.622 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2023-12-15T22:57:34.622 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T22:57:34.779 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:34.776+0000 7f01960de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:34.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:34.782+0000 7f01960de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:34.927 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:57:34.957 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:57:34.958 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-12-15T22:57:34.970 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":94,"stamp":"2023-12-15T22:57:34.607483+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":100574716,"num_objects":106,"num_object_clones":54,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":730,"num_read_kb":399948,"num_write":705,"num_write_kb":164673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":801,"ondisk_log_size":801,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":306404,"kb_used_data":199108,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377180956,"statfs":{"total":386547056640,"available":386233298944,"internally_reserved":0,"allocated":203886592,"data_stored":212492954,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1109295,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":7598,"num_write":14,"num_write_kb":4201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.020179"},"pg_stats":[{"pgid":"3.f","version":"57'14","reported_seq":135,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-12-15T22:57:29.605515+0000","last_change":"2023-12-15T22:57:29.605515+0000","last_active":"2023-12-15T22:57:29.605515+0000","last_peered":"2023-12-15T22:57:29.605515+0000","last_clean":"2023-12-15T22:57:29.605515+0000","last_became_active":"2023-12-15T22:56:39.503797+0000","last_became_peered":"2023-12-15T22:56:39.503797+0000","last_unstale":"2023-12-15T22:57:29.605515+0000","last_undegraded":"2023-12-15T22:57:29.605515+0000","last_fullsized":"2023-12-15T22:57:29.605515+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2016807,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":25,"num_read_kb":16106,"num_write":12,"num_write_kb":3599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":123,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-15T22:57:34.536879+0000","last_change":"2023-12-15T22:57:32.628493+0000","last_active":"2023-12-15T22:57:34.536879+0000","last_peered":"2023-12-15T22:57:34.536879+0000","last_clean":"2023-12-15T22:57:34.536879+0000","last_became_active":"2023-12-15T22:56:39.552856+0000","last_became_peered":"2023-12-15T22:56:39.552856+0000","last_unstale":"2023-12-15T22:57:34.536879+0000","last_undegraded":"2023-12-15T22:57:34.536879+0000","last_fullsized":"2023-12-15T22:57:34.536879+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"67'94","reported_seq":226,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-15T22:57:33.765480+0000","last_change":"2023-12-15T22:57:32.749298+0000","last_active":"2023-12-15T22:57:33.765480+0000","last_peered":"2023-12-15T22:57:33.765480+0000","last_clean":"2023-12-15T22:57:33.765480+0000","last_became_active":"2023-12-15T22:56:39.495490+0000","last_became_peered":"2023-12-15T22:56:39.495490+0000","last_unstale":"2023-12-15T22:57:33.765480+0000","last_undegraded":"2023-12-15T22:57:33.765480+0000","last_fullsized":"2023-12-15T22:57:33.765480+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075747,"num_objects":13,"num_object_clones":7,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":94,"num_read_kb":65686,"num_write":75,"num_write_kb":20912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"63'49","reported_seq":175,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-12-15T22:57:29.600155+0000","last_change":"2023-12-15T22:57:29.651777+0000","last_active":"2023-12-15T22:57:29.600155+0000","last_peered":"2023-12-15T22:57:29.600155+0000","last_clean":"2023-12-15T22:57:29.600155+0000","last_became_active":"2023-12-15T22:56:39.579234+0000","last_became_peered":"2023-12-15T22:56:39.579234+0000","last_unstale":"2023-12-15T22:57:29.600155+0000","last_undegraded":"2023-12-15T22:57:29.600155+0000","last_fullsized":"2023-12-15T22:57:29.600155+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8773221,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":55,"num_read_kb":37630,"num_write":43,"num_write_kb":11926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"35'20","reported_seq":127,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-12-15T22:57:29.695633+0000","last_change":"2023-12-15T22:57:29.704632+0000","last_active":"2023-12-15T22:57:29.695633+0000","last_peered":"2023-12-15T22:57:29.695633+0000","last_clean":"2023-12-15T22:57:29.695633+0000","last_became_active":"2023-12-15T22:56:39.572176+0000","last_became_peered":"2023-12-15T22:56:39.572176+0000","last_unstale":"2023-12-15T22:57:29.695633+0000","last_undegraded":"2023-12-15T22:57:29.695633+0000","last_fullsized":"2023-12-15T22:57:29.695633+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3310946,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":3685,"num_write":17,"num_write_kb":5025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"63'32","reported_seq":150,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-12-15T22:57:29.602081+0000","last_change":"2023-12-15T22:57:29.671451+0000","last_active":"2023-12-15T22:57:29.602081+0000","last_peered":"2023-12-15T22:57:29.602081+0000","last_clean":"2023-12-15T22:57:29.602081+0000","last_became_active":"2023-12-15T22:56:39.516361+0000","last_became_peered":"2023-12-15T22:56:39.516361+0000","last_unstale":"2023-12-15T22:57:29.602081+0000","last_undegraded":"2023-12-15T22:57:29.602081+0000","last_fullsized":"2023-12-15T22:57:29.602081+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4924418,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":19607,"num_write":31,"num_write_kb":8847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"63'148","reported_seq":257,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-12-15T22:57:29.855890+0000","last_change":"2023-12-15T22:57:29.673326+0000","last_active":"2023-12-15T22:57:29.855890+0000","last_peered":"2023-12-15T22:57:29.855890+0000","last_clean":"2023-12-15T22:57:29.855890+0000","last_became_active":"2023-12-15T22:56:39.519748+0000","last_became_peered":"2023-12-15T22:56:39.519748+0000","last_unstale":"2023-12-15T22:57:29.855890+0000","last_undegraded":"2023-12-15T22:57:29.855890+0000","last_fullsized":"2023-12-15T22:57:29.855890+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20017104,"num_objects":24,"num_object_clones":14,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":107,"num_read_kb":55715,"num_write":110,"num_write_kb":31583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"67'42","reported_seq":167,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-15T22:57:33.802340+0000","last_change":"2023-12-15T22:57:32.756668+0000","last_active":"2023-12-15T22:57:33.802340+0000","last_peered":"2023-12-15T22:57:33.802340+0000","last_clean":"2023-12-15T22:57:33.802340+0000","last_became_active":"2023-12-15T22:56:39.543179+0000","last_became_peered":"2023-12-15T22:56:39.543179+0000","last_unstale":"2023-12-15T22:57:33.802340+0000","last_undegraded":"2023-12-15T22:57:33.802340+0000","last_fullsized":"2023-12-15T22:57:33.802340+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8857223,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":38,"num_read_kb":25762,"num_write":40,"num_write_kb":10468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":216,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-12-15T22:57:29.707985+0000","last_change":"2023-12-15T22:56:32.607017+0000","last_active":"2023-12-15T22:57:29.707985+0000","last_peered":"2023-12-15T22:57:29.707985+0000","last_clean":"2023-12-15T22:57:29.707985+0000","last_became_active":"2023-12-15T22:56:32.602384+0000","last_became_peered":"2023-12-15T22:56:32.602384+0000","last_unstale":"2023-12-15T22:57:29.707985+0000","last_undegraded":"2023-12-15T22:57:29.707985+0000","last_fullsized":"2023-12-15T22:57:29.707985+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_clean_scrub_stamp":"2023-12-15T22:56:31.525572+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":97,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-15T22:57:33.640543+0000","last_change":"2023-12-15T22:56:33.627481+0000","last_active":"2023-12-15T22:57:33.640543+0000","last_peered":"2023-12-15T22:57:33.640543+0000","last_clean":"2023-12-15T22:57:33.640543+0000","last_became_active":"2023-12-15T22:56:33.624689+0000","last_became_peered":"2023-12-15T22:56:33.624689+0000","last_unstale":"2023-12-15T22:57:33.640543+0000","last_undegraded":"2023-12-15T22:57:33.640543+0000","last_fullsized":"2023-12-15T22:57:33.640543+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"64'62","reported_seq":182,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-15T22:57:33.635666+0000","last_change":"2023-12-15T22:57:32.740047+0000","last_active":"2023-12-15T22:57:33.635666+0000","last_peered":"2023-12-15T22:57:33.635666+0000","last_clean":"2023-12-15T22:57:33.635666+0000","last_became_active":"2023-12-15T22:56:39.529359+0000","last_became_peered":"2023-12-15T22:56:39.529359+0000","last_unstale":"2023-12-15T22:57:33.635666+0000","last_undegraded":"2023-12-15T22:57:33.635666+0000","last_fullsized":"2023-12-15T22:57:33.635666+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7176589,"num_objects":8,"num_object_clones":3,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":36,"num_read_kb":10704,"num_write":56,"num_write_kb":16150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":97,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-12-15T22:57:29.602053+0000","last_change":"2023-12-15T22:56:33.655797+0000","last_active":"2023-12-15T22:57:29.602053+0000","last_peered":"2023-12-15T22:57:29.602053+0000","last_clean":"2023-12-15T22:57:29.602053+0000","last_became_active":"2023-12-15T22:56:33.652885+0000","last_became_peered":"2023-12-15T22:56:33.652885+0000","last_unstale":"2023-12-15T22:57:29.602053+0000","last_undegraded":"2023-12-15T22:57:29.602053+0000","last_fullsized":"2023-12-15T22:57:29.602053+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"62'53","reported_seq":165,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-12-15T22:57:29.704658+0000","last_change":"2023-12-15T22:57:29.718984+0000","last_active":"2023-12-15T22:57:29.704658+0000","last_peered":"2023-12-15T22:57:29.704658+0000","last_clean":"2023-12-15T22:57:29.704658+0000","last_became_active":"2023-12-15T22:56:39.543509+0000","last_became_peered":"2023-12-15T22:56:39.543509+0000","last_unstale":"2023-12-15T22:57:29.704658+0000","last_undegraded":"2023-12-15T22:57:29.704658+0000","last_fullsized":"2023-12-15T22:57:29.704658+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9176556,"num_objects":9,"num_object_clones":4,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":52,"num_read_kb":33777,"num_write":39,"num_write_kb":10652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":97,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-15T22:57:33.632897+0000","last_change":"2023-12-15T22:56:33.600485+0000","last_active":"2023-12-15T22:57:33.632897+0000","last_peered":"2023-12-15T22:57:33.632897+0000","last_clean":"2023-12-15T22:57:33.632897+0000","last_became_active":"2023-12-15T22:56:33.597320+0000","last_became_peered":"2023-12-15T22:56:33.597320+0000","last_unstale":"2023-12-15T22:57:33.632897+0000","last_undegraded":"2023-12-15T22:57:33.632897+0000","last_fullsized":"2023-12-15T22:57:33.632897+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"55'26","reported_seq":133,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-12-15T22:57:29.708088+0000","last_change":"2023-12-15T22:57:29.716266+0000","last_active":"2023-12-15T22:57:29.708088+0000","last_peered":"2023-12-15T22:57:29.708088+0000","last_clean":"2023-12-15T22:57:29.708088+0000","last_became_active":"2023-12-15T22:56:39.560980+0000","last_became_peered":"2023-12-15T22:56:39.560980+0000","last_unstale":"2023-12-15T22:57:29.708088+0000","last_undegraded":"2023-12-15T22:57:29.708088+0000","last_fullsized":"2023-12-15T22:57:29.708088+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5747961,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":13740,"num_write":21,"num_write_kb":6063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":97,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-12-15T22:57:29.604444+0000","last_change":"2023-12-15T22:56:33.648541+0000","last_active":"2023-12-15T22:57:29.604444+0000","last_peered":"2023-12-15T22:57:29.604444+0000","last_clean":"2023-12-15T22:57:29.604444+0000","last_became_active":"2023-12-15T22:56:33.643681+0000","last_became_peered":"2023-12-15T22:56:33.643681+0000","last_unstale":"2023-12-15T22:57:29.604444+0000","last_undegraded":"2023-12-15T22:57:29.604444+0000","last_fullsized":"2023-12-15T22:57:29.604444+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"61'57","reported_seq":190,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-15T22:57:34.543535+0000","last_change":"2023-12-15T22:57:32.628785+0000","last_active":"2023-12-15T22:57:34.543535+0000","last_peered":"2023-12-15T22:57:34.543535+0000","last_clean":"2023-12-15T22:57:34.543535+0000","last_became_active":"2023-12-15T22:56:39.506510+0000","last_became_peered":"2023-12-15T22:56:39.506510+0000","last_unstale":"2023-12-15T22:57:34.543535+0000","last_undegraded":"2023-12-15T22:57:34.543535+0000","last_fullsized":"2023-12-15T22:57:34.543535+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8789995,"num_objects":10,"num_object_clones":5,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":59,"num_read_kb":42594,"num_write":44,"num_write_kb":13017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"50'13","reported_seq":125,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-15T22:57:33.636744+0000","last_change":"2023-12-15T22:57:32.749046+0000","last_active":"2023-12-15T22:57:33.636744+0000","last_peered":"2023-12-15T22:57:33.636744+0000","last_clean":"2023-12-15T22:57:33.636744+0000","last_became_active":"2023-12-15T22:56:39.511159+0000","last_became_peered":"2023-12-15T22:56:39.511159+0000","last_unstale":"2023-12-15T22:57:33.636744+0000","last_undegraded":"2023-12-15T22:57:33.636744+0000","last_fullsized":"2023-12-15T22:57:33.636744+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":5,"num_read_kb":2,"num_write":9,"num_write_kb":2133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":103,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-15T22:57:34.533926+0000","last_change":"2023-12-15T22:56:33.633890+0000","last_active":"2023-12-15T22:57:34.533926+0000","last_peered":"2023-12-15T22:57:34.533926+0000","last_clean":"2023-12-15T22:57:34.533926+0000","last_became_active":"2023-12-15T22:56:33.631134+0000","last_became_peered":"2023-12-15T22:56:33.631134+0000","last_unstale":"2023-12-15T22:57:34.533926+0000","last_undegraded":"2023-12-15T22:57:34.533926+0000","last_fullsized":"2023-12-15T22:57:34.533926+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"54'47","reported_seq":163,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-15T22:57:34.542803+0000","last_change":"2023-12-15T22:57:32.624144+0000","last_active":"2023-12-15T22:57:34.542803+0000","last_peered":"2023-12-15T22:57:34.542803+0000","last_clean":"2023-12-15T22:57:34.542803+0000","last_became_active":"2023-12-15T22:56:39.532730+0000","last_became_peered":"2023-12-15T22:56:39.532730+0000","last_unstale":"2023-12-15T22:57:34.542803+0000","last_undegraded":"2023-12-15T22:57:34.542803+0000","last_fullsized":"2023-12-15T22:57:34.542803+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3764735,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":35,"num_read_kb":28298,"num_write":27,"num_write_kb":8830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":87,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-12-15T22:57:29.699507+0000","last_change":"2023-12-15T22:56:33.655736+0000","last_active":"2023-12-15T22:57:29.699507+0000","last_peered":"2023-12-15T22:57:29.699507+0000","last_clean":"2023-12-15T22:57:29.699507+0000","last_became_active":"2023-12-15T22:56:33.652917+0000","last_became_peered":"2023-12-15T22:56:33.652917+0000","last_unstale":"2023-12-15T22:57:29.699507+0000","last_undegraded":"2023-12-15T22:57:29.699507+0000","last_fullsized":"2023-12-15T22:57:29.699507+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"37'40","reported_seq":162,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-12-15T22:57:29.713546+0000","last_change":"2023-12-15T22:57:29.717038+0000","last_active":"2023-12-15T22:57:29.713546+0000","last_peered":"2023-12-15T22:57:29.713546+0000","last_clean":"2023-12-15T22:57:29.713546+0000","last_became_active":"2023-12-15T22:56:39.557252+0000","last_became_peered":"2023-12-15T22:56:39.557252+0000","last_unstale":"2023-12-15T22:57:29.713546+0000","last_undegraded":"2023-12-15T22:57:29.713546+0000","last_fullsized":"2023-12-15T22:57:29.713546+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5596698,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":51,"num_read_kb":39064,"num_write":36,"num_write_kb":11126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":103,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-15T22:57:34.539537+0000","last_change":"2023-12-15T22:56:33.632348+0000","last_active":"2023-12-15T22:57:34.539537+0000","last_peered":"2023-12-15T22:57:34.539537+0000","last_clean":"2023-12-15T22:57:34.539537+0000","last_became_active":"2023-12-15T22:56:33.629022+0000","last_became_peered":"2023-12-15T22:56:33.629022+0000","last_unstale":"2023-12-15T22:57:34.539537+0000","last_undegraded":"2023-12-15T22:57:34.539537+0000","last_fullsized":"2023-12-15T22:57:34.539537+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"64'7","reported_seq":134,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-15T22:57:34.536133+0000","last_change":"2023-12-15T22:57:32.628314+0000","last_active":"2023-12-15T22:57:34.536133+0000","last_peered":"2023-12-15T22:57:34.536133+0000","last_clean":"2023-12-15T22:57:34.536133+0000","last_became_active":"2023-12-15T22:56:39.535145+0000","last_became_peered":"2023-12-15T22:56:39.535145+0000","last_unstale":"2023-12-15T22:57:34.536133+0000","last_undegraded":"2023-12-15T22:57:34.536133+0000","last_fullsized":"2023-12-15T22:57:34.536133+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2756348,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":7487,"num_write":8,"num_write_kb":2448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":97,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-15T22:57:33.633436+0000","last_change":"2023-12-15T22:56:33.627005+0000","last_active":"2023-12-15T22:57:33.633436+0000","last_peered":"2023-12-15T22:57:33.633436+0000","last_clean":"2023-12-15T22:57:33.633436+0000","last_became_active":"2023-12-15T22:56:33.622181+0000","last_became_peered":"2023-12-15T22:56:33.622181+0000","last_unstale":"2023-12-15T22:57:33.633436+0000","last_undegraded":"2023-12-15T22:57:33.633436+0000","last_fullsized":"2023-12-15T22:57:33.633436+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":99984348,"num_objects":104,"num_object_clones":54,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":627,"num_read_kb":399857,"num_write":568,"num_write_kb":162779,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":704,"ondisk_log_size":704,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705724,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75472,"kb_used_data":48648,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94296368,"statfs":{"total":96636764160,"available":96559480832,"internally_reserved":0,"allocated":49815552,"data_stored":50440026,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705728,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":72116,"kb_used_data":45292,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94299724,"statfs":{"total":96636764160,"available":96562917376,"internally_reserved":0,"allocated":46379008,"data_stored":50127435,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705728,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":76000,"kb_used_data":49176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94295840,"statfs":{"total":96636764160,"available":96558940160,"internally_reserved":0,"allocated":50356224,"data_stored":52580801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705724,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":82816,"kb_used_data":55992,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94289024,"statfs":{"total":96636764160,"available":96551960576,"internally_reserved":0,"allocated":57335808,"data_stored":59344692,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-15T22:57:34.971 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T22:57:35.032 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:35.031+0000 7fb173a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:35.036 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:35.036+0000 7fb173a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:35.120 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:35.117+0000 7efd1211e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:35.123 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:35.123+0000 7efd1211e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:35.298 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:57:35.298 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-12-15T22:57:35.310 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":94,"stamp":"2023-12-15T22:57:34.607483+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":100574716,"num_objects":106,"num_object_clones":54,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":730,"num_read_kb":399948,"num_write":705,"num_write_kb":164673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":801,"ondisk_log_size":801,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":306404,"kb_used_data":199108,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377180956,"statfs":{"total":386547056640,"available":386233298944,"internally_reserved":0,"allocated":203886592,"data_stored":212492954,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1109295,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":7598,"num_write":14,"num_write_kb":4201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.020179"},"pg_stats":[{"pgid":"3.f","version":"57'14","reported_seq":135,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-12-15T22:57:29.605515+0000","last_change":"2023-12-15T22:57:29.605515+0000","last_active":"2023-12-15T22:57:29.605515+0000","last_peered":"2023-12-15T22:57:29.605515+0000","last_clean":"2023-12-15T22:57:29.605515+0000","last_became_active":"2023-12-15T22:56:39.503797+0000","last_became_peered":"2023-12-15T22:56:39.503797+0000","last_unstale":"2023-12-15T22:57:29.605515+0000","last_undegraded":"2023-12-15T22:57:29.605515+0000","last_fullsized":"2023-12-15T22:57:29.605515+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2016807,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":25,"num_read_kb":16106,"num_write":12,"num_write_kb":3599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":123,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-15T22:57:34.536879+0000","last_change":"2023-12-15T22:57:32.628493+0000","last_active":"2023-12-15T22:57:34.536879+0000","last_peered":"2023-12-15T22:57:34.536879+0000","last_clean":"2023-12-15T22:57:34.536879+0000","last_became_active":"2023-12-15T22:56:39.552856+0000","last_became_peered":"2023-12-15T22:56:39.552856+0000","last_unstale":"2023-12-15T22:57:34.536879+0000","last_undegraded":"2023-12-15T22:57:34.536879+0000","last_fullsized":"2023-12-15T22:57:34.536879+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"67'94","reported_seq":226,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-15T22:57:33.765480+0000","last_change":"2023-12-15T22:57:32.749298+0000","last_active":"2023-12-15T22:57:33.765480+0000","last_peered":"2023-12-15T22:57:33.765480+0000","last_clean":"2023-12-15T22:57:33.765480+0000","last_became_active":"2023-12-15T22:56:39.495490+0000","last_became_peered":"2023-12-15T22:56:39.495490+0000","last_unstale":"2023-12-15T22:57:33.765480+0000","last_undegraded":"2023-12-15T22:57:33.765480+0000","last_fullsized":"2023-12-15T22:57:33.765480+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075747,"num_objects":13,"num_object_clones":7,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":94,"num_read_kb":65686,"num_write":75,"num_write_kb":20912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"63'49","reported_seq":175,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-12-15T22:57:29.600155+0000","last_change":"2023-12-15T22:57:29.651777+0000","last_active":"2023-12-15T22:57:29.600155+0000","last_peered":"2023-12-15T22:57:29.600155+0000","last_clean":"2023-12-15T22:57:29.600155+0000","last_became_active":"2023-12-15T22:56:39.579234+0000","last_became_peered":"2023-12-15T22:56:39.579234+0000","last_unstale":"2023-12-15T22:57:29.600155+0000","last_undegraded":"2023-12-15T22:57:29.600155+0000","last_fullsized":"2023-12-15T22:57:29.600155+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8773221,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":55,"num_read_kb":37630,"num_write":43,"num_write_kb":11926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"35'20","reported_seq":127,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-12-15T22:57:29.695633+0000","last_change":"2023-12-15T22:57:29.704632+0000","last_active":"2023-12-15T22:57:29.695633+0000","last_peered":"2023-12-15T22:57:29.695633+0000","last_clean":"2023-12-15T22:57:29.695633+0000","last_became_active":"2023-12-15T22:56:39.572176+0000","last_became_peered":"2023-12-15T22:56:39.572176+0000","last_unstale":"2023-12-15T22:57:29.695633+0000","last_undegraded":"2023-12-15T22:57:29.695633+0000","last_fullsized":"2023-12-15T22:57:29.695633+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3310946,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":3685,"num_write":17,"num_write_kb":5025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"63'32","reported_seq":150,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-12-15T22:57:29.602081+0000","last_change":"2023-12-15T22:57:29.671451+0000","last_active":"2023-12-15T22:57:29.602081+0000","last_peered":"2023-12-15T22:57:29.602081+0000","last_clean":"2023-12-15T22:57:29.602081+0000","last_became_active":"2023-12-15T22:56:39.516361+0000","last_became_peered":"2023-12-15T22:56:39.516361+0000","last_unstale":"2023-12-15T22:57:29.602081+0000","last_undegraded":"2023-12-15T22:57:29.602081+0000","last_fullsized":"2023-12-15T22:57:29.602081+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4924418,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":19607,"num_write":31,"num_write_kb":8847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"63'148","reported_seq":257,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-12-15T22:57:29.855890+0000","last_change":"2023-12-15T22:57:29.673326+0000","last_active":"2023-12-15T22:57:29.855890+0000","last_peered":"2023-12-15T22:57:29.855890+0000","last_clean":"2023-12-15T22:57:29.855890+0000","last_became_active":"2023-12-15T22:56:39.519748+0000","last_became_peered":"2023-12-15T22:56:39.519748+0000","last_unstale":"2023-12-15T22:57:29.855890+0000","last_undegraded":"2023-12-15T22:57:29.855890+0000","last_fullsized":"2023-12-15T22:57:29.855890+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20017104,"num_objects":24,"num_object_clones":14,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":107,"num_read_kb":55715,"num_write":110,"num_write_kb":31583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"67'42","reported_seq":167,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-15T22:57:33.802340+0000","last_change":"2023-12-15T22:57:32.756668+0000","last_active":"2023-12-15T22:57:33.802340+0000","last_peered":"2023-12-15T22:57:33.802340+0000","last_clean":"2023-12-15T22:57:33.802340+0000","last_became_active":"2023-12-15T22:56:39.543179+0000","last_became_peered":"2023-12-15T22:56:39.543179+0000","last_unstale":"2023-12-15T22:57:33.802340+0000","last_undegraded":"2023-12-15T22:57:33.802340+0000","last_fullsized":"2023-12-15T22:57:33.802340+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8857223,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":38,"num_read_kb":25762,"num_write":40,"num_write_kb":10468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":216,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-12-15T22:57:29.707985+0000","last_change":"2023-12-15T22:56:32.607017+0000","last_active":"2023-12-15T22:57:29.707985+0000","last_peered":"2023-12-15T22:57:29.707985+0000","last_clean":"2023-12-15T22:57:29.707985+0000","last_became_active":"2023-12-15T22:56:32.602384+0000","last_became_peered":"2023-12-15T22:56:32.602384+0000","last_unstale":"2023-12-15T22:57:29.707985+0000","last_undegraded":"2023-12-15T22:57:29.707985+0000","last_fullsized":"2023-12-15T22:57:29.707985+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_clean_scrub_stamp":"2023-12-15T22:56:31.525572+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":97,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-15T22:57:33.640543+0000","last_change":"2023-12-15T22:56:33.627481+0000","last_active":"2023-12-15T22:57:33.640543+0000","last_peered":"2023-12-15T22:57:33.640543+0000","last_clean":"2023-12-15T22:57:33.640543+0000","last_became_active":"2023-12-15T22:56:33.624689+0000","last_became_peered":"2023-12-15T22:56:33.624689+0000","last_unstale":"2023-12-15T22:57:33.640543+0000","last_undegraded":"2023-12-15T22:57:33.640543+0000","last_fullsized":"2023-12-15T22:57:33.640543+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"64'62","reported_seq":182,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-15T22:57:33.635666+0000","last_change":"2023-12-15T22:57:32.740047+0000","last_active":"2023-12-15T22:57:33.635666+0000","last_peered":"2023-12-15T22:57:33.635666+0000","last_clean":"2023-12-15T22:57:33.635666+0000","last_became_active":"2023-12-15T22:56:39.529359+0000","last_became_peered":"2023-12-15T22:56:39.529359+0000","last_unstale":"2023-12-15T22:57:33.635666+0000","last_undegraded":"2023-12-15T22:57:33.635666+0000","last_fullsized":"2023-12-15T22:57:33.635666+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7176589,"num_objects":8,"num_object_clones":3,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":36,"num_read_kb":10704,"num_write":56,"num_write_kb":16150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":97,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-12-15T22:57:29.602053+0000","last_change":"2023-12-15T22:56:33.655797+0000","last_active":"2023-12-15T22:57:29.602053+0000","last_peered":"2023-12-15T22:57:29.602053+0000","last_clean":"2023-12-15T22:57:29.602053+0000","last_became_active":"2023-12-15T22:56:33.652885+0000","last_became_peered":"2023-12-15T22:56:33.652885+0000","last_unstale":"2023-12-15T22:57:29.602053+0000","last_undegraded":"2023-12-15T22:57:29.602053+0000","last_fullsized":"2023-12-15T22:57:29.602053+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"62'53","reported_seq":165,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-12-15T22:57:29.704658+0000","last_change":"2023-12-15T22:57:29.718984+0000","last_active":"2023-12-15T22:57:29.704658+0000","last_peered":"2023-12-15T22:57:29.704658+0000","last_clean":"2023-12-15T22:57:29.704658+0000","last_became_active":"2023-12-15T22:56:39.543509+0000","last_became_peered":"2023-12-15T22:56:39.543509+0000","last_unstale":"2023-12-15T22:57:29.704658+0000","last_undegraded":"2023-12-15T22:57:29.704658+0000","last_fullsized":"2023-12-15T22:57:29.704658+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9176556,"num_objects":9,"num_object_clones":4,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":52,"num_read_kb":33777,"num_write":39,"num_write_kb":10652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":97,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-15T22:57:33.632897+0000","last_change":"2023-12-15T22:56:33.600485+0000","last_active":"2023-12-15T22:57:33.632897+0000","last_peered":"2023-12-15T22:57:33.632897+0000","last_clean":"2023-12-15T22:57:33.632897+0000","last_became_active":"2023-12-15T22:56:33.597320+0000","last_became_peered":"2023-12-15T22:56:33.597320+0000","last_unstale":"2023-12-15T22:57:33.632897+0000","last_undegraded":"2023-12-15T22:57:33.632897+0000","last_fullsized":"2023-12-15T22:57:33.632897+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"55'26","reported_seq":133,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-12-15T22:57:29.708088+0000","last_change":"2023-12-15T22:57:29.716266+0000","last_active":"2023-12-15T22:57:29.708088+0000","last_peered":"2023-12-15T22:57:29.708088+0000","last_clean":"2023-12-15T22:57:29.708088+0000","last_became_active":"2023-12-15T22:56:39.560980+0000","last_became_peered":"2023-12-15T22:56:39.560980+0000","last_unstale":"2023-12-15T22:57:29.708088+0000","last_undegraded":"2023-12-15T22:57:29.708088+0000","last_fullsized":"2023-12-15T22:57:29.708088+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5747961,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":13740,"num_write":21,"num_write_kb":6063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":97,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-12-15T22:57:29.604444+0000","last_change":"2023-12-15T22:56:33.648541+0000","last_active":"2023-12-15T22:57:29.604444+0000","last_peered":"2023-12-15T22:57:29.604444+0000","last_clean":"2023-12-15T22:57:29.604444+0000","last_became_active":"2023-12-15T22:56:33.643681+0000","last_became_peered":"2023-12-15T22:56:33.643681+0000","last_unstale":"2023-12-15T22:57:29.604444+0000","last_undegraded":"2023-12-15T22:57:29.604444+0000","last_fullsized":"2023-12-15T22:57:29.604444+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"61'57","reported_seq":190,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-15T22:57:34.543535+0000","last_change":"2023-12-15T22:57:32.628785+0000","last_active":"2023-12-15T22:57:34.543535+0000","last_peered":"2023-12-15T22:57:34.543535+0000","last_clean":"2023-12-15T22:57:34.543535+0000","last_became_active":"2023-12-15T22:56:39.506510+0000","last_became_peered":"2023-12-15T22:56:39.506510+0000","last_unstale":"2023-12-15T22:57:34.543535+0000","last_undegraded":"2023-12-15T22:57:34.543535+0000","last_fullsized":"2023-12-15T22:57:34.543535+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8789995,"num_objects":10,"num_object_clones":5,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":59,"num_read_kb":42594,"num_write":44,"num_write_kb":13017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"50'13","reported_seq":125,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-15T22:57:33.636744+0000","last_change":"2023-12-15T22:57:32.749046+0000","last_active":"2023-12-15T22:57:33.636744+0000","last_peered":"2023-12-15T22:57:33.636744+0000","last_clean":"2023-12-15T22:57:33.636744+0000","last_became_active":"2023-12-15T22:56:39.511159+0000","last_became_peered":"2023-12-15T22:56:39.511159+0000","last_unstale":"2023-12-15T22:57:33.636744+0000","last_undegraded":"2023-12-15T22:57:33.636744+0000","last_fullsized":"2023-12-15T22:57:33.636744+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":5,"num_read_kb":2,"num_write":9,"num_write_kb":2133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":103,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-15T22:57:34.533926+0000","last_change":"2023-12-15T22:56:33.633890+0000","last_active":"2023-12-15T22:57:34.533926+0000","last_peered":"2023-12-15T22:57:34.533926+0000","last_clean":"2023-12-15T22:57:34.533926+0000","last_became_active":"2023-12-15T22:56:33.631134+0000","last_became_peered":"2023-12-15T22:56:33.631134+0000","last_unstale":"2023-12-15T22:57:34.533926+0000","last_undegraded":"2023-12-15T22:57:34.533926+0000","last_fullsized":"2023-12-15T22:57:34.533926+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"54'47","reported_seq":163,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-15T22:57:34.542803+0000","last_change":"2023-12-15T22:57:32.624144+0000","last_active":"2023-12-15T22:57:34.542803+0000","last_peered":"2023-12-15T22:57:34.542803+0000","last_clean":"2023-12-15T22:57:34.542803+0000","last_became_active":"2023-12-15T22:56:39.532730+0000","last_became_peered":"2023-12-15T22:56:39.532730+0000","last_unstale":"2023-12-15T22:57:34.542803+0000","last_undegraded":"2023-12-15T22:57:34.542803+0000","last_fullsized":"2023-12-15T22:57:34.542803+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3764735,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":35,"num_read_kb":28298,"num_write":27,"num_write_kb":8830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":87,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-12-15T22:57:29.699507+0000","last_change":"2023-12-15T22:56:33.655736+0000","last_active":"2023-12-15T22:57:29.699507+0000","last_peered":"2023-12-15T22:57:29.699507+0000","last_clean":"2023-12-15T22:57:29.699507+0000","last_became_active":"2023-12-15T22:56:33.652917+0000","last_became_peered":"2023-12-15T22:56:33.652917+0000","last_unstale":"2023-12-15T22:57:29.699507+0000","last_undegraded":"2023-12-15T22:57:29.699507+0000","last_fullsized":"2023-12-15T22:57:29.699507+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"37'40","reported_seq":162,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-12-15T22:57:29.713546+0000","last_change":"2023-12-15T22:57:29.717038+0000","last_active":"2023-12-15T22:57:29.713546+0000","last_peered":"2023-12-15T22:57:29.713546+0000","last_clean":"2023-12-15T22:57:29.713546+0000","last_became_active":"2023-12-15T22:56:39.557252+0000","last_became_peered":"2023-12-15T22:56:39.557252+0000","last_unstale":"2023-12-15T22:57:29.713546+0000","last_undegraded":"2023-12-15T22:57:29.713546+0000","last_fullsized":"2023-12-15T22:57:29.713546+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5596698,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":51,"num_read_kb":39064,"num_write":36,"num_write_kb":11126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":103,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-15T22:57:34.539537+0000","last_change":"2023-12-15T22:56:33.632348+0000","last_active":"2023-12-15T22:57:34.539537+0000","last_peered":"2023-12-15T22:57:34.539537+0000","last_clean":"2023-12-15T22:57:34.539537+0000","last_became_active":"2023-12-15T22:56:33.629022+0000","last_became_peered":"2023-12-15T22:56:33.629022+0000","last_unstale":"2023-12-15T22:57:34.539537+0000","last_undegraded":"2023-12-15T22:57:34.539537+0000","last_fullsized":"2023-12-15T22:57:34.539537+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"64'7","reported_seq":134,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-15T22:57:34.536133+0000","last_change":"2023-12-15T22:57:32.628314+0000","last_active":"2023-12-15T22:57:34.536133+0000","last_peered":"2023-12-15T22:57:34.536133+0000","last_clean":"2023-12-15T22:57:34.536133+0000","last_became_active":"2023-12-15T22:56:39.535145+0000","last_became_peered":"2023-12-15T22:56:39.535145+0000","last_unstale":"2023-12-15T22:57:34.536133+0000","last_undegraded":"2023-12-15T22:57:34.536133+0000","last_fullsized":"2023-12-15T22:57:34.536133+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2756348,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":7487,"num_write":8,"num_write_kb":2448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":97,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-15T22:57:33.633436+0000","last_change":"2023-12-15T22:56:33.627005+0000","last_active":"2023-12-15T22:57:33.633436+0000","last_peered":"2023-12-15T22:57:33.633436+0000","last_clean":"2023-12-15T22:57:33.633436+0000","last_became_active":"2023-12-15T22:56:33.622181+0000","last_became_peered":"2023-12-15T22:56:33.622181+0000","last_unstale":"2023-12-15T22:57:33.633436+0000","last_undegraded":"2023-12-15T22:57:33.633436+0000","last_fullsized":"2023-12-15T22:57:33.633436+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":99984348,"num_objects":104,"num_object_clones":54,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":627,"num_read_kb":399857,"num_write":568,"num_write_kb":162779,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":704,"ondisk_log_size":704,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705724,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75472,"kb_used_data":48648,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94296368,"statfs":{"total":96636764160,"available":96559480832,"internally_reserved":0,"allocated":49815552,"data_stored":50440026,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705728,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":72116,"kb_used_data":45292,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94299724,"statfs":{"total":96636764160,"available":96562917376,"internally_reserved":0,"allocated":46379008,"data_stored":50127435,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705728,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":76000,"kb_used_data":49176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94295840,"statfs":{"total":96636764160,"available":96558940160,"internally_reserved":0,"allocated":50356224,"data_stored":52580801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705724,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":82816,"kb_used_data":55992,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94289024,"statfs":{"total":96636764160,"available":96551960576,"internally_reserved":0,"allocated":57335808,"data_stored":59344692,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-15T22:57:35.311 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2023-12-15T22:57:35.311 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T22:57:35.312 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T22:57:35.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:35.460+0000 7fef8a217640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:35.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:35.466+0000 7fef8a217640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:35.610 INFO:tasks.rados.rados.0.smithi071.stdout:299: done (1 left) 2023-12-15T22:57:35.610 INFO:tasks.rados.rados.0.smithi071.stdout:300: read oid 6 snap -1 2023-12-15T22:57:35.610 INFO:tasks.rados.rados.0.smithi071.stdout:300: expect (ObjNum 66 snap 3 seq_num 66) 2023-12-15T22:57:35.610 INFO:tasks.rados.rados.0.smithi071.stdout:301: snap_remove snap 26 2023-12-15T22:57:35.731 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:57:35.731 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-12-15T22:57:35.744 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2023-12-15T22:57:35.608819+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":101549522,"num_objects":110,"num_object_clones":57,"num_object_copies":330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":746,"num_read_kb":411994,"num_write":719,"num_write_kb":167952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":831,"ondisk_log_size":831,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":313160,"kb_used_data":205864,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377174200,"statfs":{"total":386547056640,"available":386226380800,"internally_reserved":0,"allocated":210804736,"data_stored":225527462,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2084101,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":26,"num_read_kb":19644,"num_write":28,"num_write_kb":7480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.020561"},"pg_stats":[{"pgid":"3.f","version":"66'18","reported_seq":143,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-12-15T22:57:35.521892+0000","last_change":"2023-12-15T22:57:33.642213+0000","last_active":"2023-12-15T22:57:35.521892+0000","last_peered":"2023-12-15T22:57:35.521892+0000","last_clean":"2023-12-15T22:57:35.521892+0000","last_became_active":"2023-12-15T22:56:39.503797+0000","last_became_peered":"2023-12-15T22:56:39.503797+0000","last_unstale":"2023-12-15T22:57:35.521892+0000","last_undegraded":"2023-12-15T22:57:35.521892+0000","last_fullsized":"2023-12-15T22:57:35.521892+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2016807,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":25,"num_read_kb":16106,"num_write":12,"num_write_kb":3599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":123,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-15T22:57:34.536879+0000","last_change":"2023-12-15T22:57:32.628493+0000","last_active":"2023-12-15T22:57:34.536879+0000","last_peered":"2023-12-15T22:57:34.536879+0000","last_clean":"2023-12-15T22:57:34.536879+0000","last_became_active":"2023-12-15T22:56:39.552856+0000","last_became_peered":"2023-12-15T22:56:39.552856+0000","last_unstale":"2023-12-15T22:57:34.536879+0000","last_undegraded":"2023-12-15T22:57:34.536879+0000","last_fullsized":"2023-12-15T22:57:34.536879+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"67'94","reported_seq":226,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-15T22:57:33.765480+0000","last_change":"2023-12-15T22:57:32.749298+0000","last_active":"2023-12-15T22:57:33.765480+0000","last_peered":"2023-12-15T22:57:33.765480+0000","last_clean":"2023-12-15T22:57:33.765480+0000","last_became_active":"2023-12-15T22:56:39.495490+0000","last_became_peered":"2023-12-15T22:56:39.495490+0000","last_unstale":"2023-12-15T22:57:33.765480+0000","last_undegraded":"2023-12-15T22:57:33.765480+0000","last_fullsized":"2023-12-15T22:57:33.765480+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075747,"num_objects":13,"num_object_clones":7,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":94,"num_read_kb":65686,"num_write":75,"num_write_kb":20912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"67'61","reported_seq":193,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-12-15T22:57:35.534808+0000","last_change":"2023-12-15T22:57:33.653945+0000","last_active":"2023-12-15T22:57:35.534808+0000","last_peered":"2023-12-15T22:57:35.534808+0000","last_clean":"2023-12-15T22:57:35.534808+0000","last_became_active":"2023-12-15T22:56:39.579234+0000","last_became_peered":"2023-12-15T22:56:39.579234+0000","last_unstale":"2023-12-15T22:57:35.534808+0000","last_undegraded":"2023-12-15T22:57:35.534808+0000","last_fullsized":"2023-12-15T22:57:35.534808+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9025326,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":57,"num_read_kb":37632,"num_write":53,"num_write_kb":14095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"64'23","reported_seq":143,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-12-15T22:57:35.524170+0000","last_change":"2023-12-15T22:57:32.636891+0000","last_active":"2023-12-15T22:57:35.524170+0000","last_peered":"2023-12-15T22:57:35.524170+0000","last_clean":"2023-12-15T22:57:35.524170+0000","last_became_active":"2023-12-15T22:56:39.572176+0000","last_became_peered":"2023-12-15T22:56:39.572176+0000","last_unstale":"2023-12-15T22:57:35.524170+0000","last_undegraded":"2023-12-15T22:57:35.524170+0000","last_fullsized":"2023-12-15T22:57:35.524170+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6050487,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":3686,"num_write":21,"num_write_kb":6135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"66'34","reported_seq":158,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-12-15T22:57:35.522732+0000","last_change":"2023-12-15T22:57:33.642867+0000","last_active":"2023-12-15T22:57:35.522732+0000","last_peered":"2023-12-15T22:57:35.522732+0000","last_clean":"2023-12-15T22:57:35.522732+0000","last_became_active":"2023-12-15T22:56:39.516361+0000","last_became_peered":"2023-12-15T22:56:39.516361+0000","last_unstale":"2023-12-15T22:57:35.522732+0000","last_undegraded":"2023-12-15T22:57:35.522732+0000","last_fullsized":"2023-12-15T22:57:35.522732+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4924418,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":19607,"num_write":31,"num_write_kb":8847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"67'154","reported_seq":270,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-12-15T22:57:35.519658+0000","last_change":"2023-12-15T22:57:33.642515+0000","last_active":"2023-12-15T22:57:35.519658+0000","last_peered":"2023-12-15T22:57:35.519658+0000","last_clean":"2023-12-15T22:57:35.519658+0000","last_became_active":"2023-12-15T22:56:39.519748+0000","last_became_peered":"2023-12-15T22:56:39.519748+0000","last_unstale":"2023-12-15T22:57:35.519658+0000","last_undegraded":"2023-12-15T22:57:35.519658+0000","last_fullsized":"2023-12-15T22:57:35.519658+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20017104,"num_objects":24,"num_object_clones":14,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":119,"num_read_kb":67758,"num_write":110,"num_write_kb":31583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"67'42","reported_seq":167,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-15T22:57:33.802340+0000","last_change":"2023-12-15T22:57:32.756668+0000","last_active":"2023-12-15T22:57:33.802340+0000","last_peered":"2023-12-15T22:57:33.802340+0000","last_clean":"2023-12-15T22:57:33.802340+0000","last_became_active":"2023-12-15T22:56:39.543179+0000","last_became_peered":"2023-12-15T22:56:39.543179+0000","last_unstale":"2023-12-15T22:57:33.802340+0000","last_undegraded":"2023-12-15T22:57:33.802340+0000","last_fullsized":"2023-12-15T22:57:33.802340+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8857223,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":38,"num_read_kb":25762,"num_write":40,"num_write_kb":10468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":226,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-12-15T22:57:35.526074+0000","last_change":"2023-12-15T22:56:32.607017+0000","last_active":"2023-12-15T22:57:35.526074+0000","last_peered":"2023-12-15T22:57:35.526074+0000","last_clean":"2023-12-15T22:57:35.526074+0000","last_became_active":"2023-12-15T22:56:32.602384+0000","last_became_peered":"2023-12-15T22:56:32.602384+0000","last_unstale":"2023-12-15T22:57:35.526074+0000","last_undegraded":"2023-12-15T22:57:35.526074+0000","last_fullsized":"2023-12-15T22:57:35.526074+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_clean_scrub_stamp":"2023-12-15T22:56:31.525572+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":97,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-15T22:57:33.640543+0000","last_change":"2023-12-15T22:56:33.627481+0000","last_active":"2023-12-15T22:57:33.640543+0000","last_peered":"2023-12-15T22:57:33.640543+0000","last_clean":"2023-12-15T22:57:33.640543+0000","last_became_active":"2023-12-15T22:56:33.624689+0000","last_became_peered":"2023-12-15T22:56:33.624689+0000","last_unstale":"2023-12-15T22:57:33.640543+0000","last_undegraded":"2023-12-15T22:57:33.640543+0000","last_fullsized":"2023-12-15T22:57:33.640543+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"64'62","reported_seq":182,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-15T22:57:33.635666+0000","last_change":"2023-12-15T22:57:32.740047+0000","last_active":"2023-12-15T22:57:33.635666+0000","last_peered":"2023-12-15T22:57:33.635666+0000","last_clean":"2023-12-15T22:57:33.635666+0000","last_became_active":"2023-12-15T22:56:39.529359+0000","last_became_peered":"2023-12-15T22:56:39.529359+0000","last_unstale":"2023-12-15T22:57:33.635666+0000","last_undegraded":"2023-12-15T22:57:33.635666+0000","last_fullsized":"2023-12-15T22:57:33.635666+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7176589,"num_objects":8,"num_object_clones":3,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":36,"num_read_kb":10704,"num_write":56,"num_write_kb":16150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":103,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-12-15T22:57:35.524456+0000","last_change":"2023-12-15T22:56:33.655797+0000","last_active":"2023-12-15T22:57:35.524456+0000","last_peered":"2023-12-15T22:57:35.524456+0000","last_clean":"2023-12-15T22:57:35.524456+0000","last_became_active":"2023-12-15T22:56:33.652885+0000","last_became_peered":"2023-12-15T22:56:33.652885+0000","last_unstale":"2023-12-15T22:57:35.524456+0000","last_undegraded":"2023-12-15T22:57:35.524456+0000","last_fullsized":"2023-12-15T22:57:35.524456+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"62'53","reported_seq":177,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-12-15T22:57:35.522195+0000","last_change":"2023-12-15T22:57:32.640556+0000","last_active":"2023-12-15T22:57:35.522195+0000","last_peered":"2023-12-15T22:57:35.522195+0000","last_clean":"2023-12-15T22:57:35.522195+0000","last_became_active":"2023-12-15T22:56:39.543509+0000","last_became_peered":"2023-12-15T22:56:39.543509+0000","last_unstale":"2023-12-15T22:57:35.522195+0000","last_undegraded":"2023-12-15T22:57:35.522195+0000","last_fullsized":"2023-12-15T22:57:35.522195+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9176556,"num_objects":9,"num_object_clones":4,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":52,"num_read_kb":33777,"num_write":39,"num_write_kb":10652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":97,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-15T22:57:33.632897+0000","last_change":"2023-12-15T22:56:33.600485+0000","last_active":"2023-12-15T22:57:33.632897+0000","last_peered":"2023-12-15T22:57:33.632897+0000","last_clean":"2023-12-15T22:57:33.632897+0000","last_became_active":"2023-12-15T22:56:33.597320+0000","last_became_peered":"2023-12-15T22:56:33.597320+0000","last_unstale":"2023-12-15T22:57:33.632897+0000","last_undegraded":"2023-12-15T22:57:33.632897+0000","last_fullsized":"2023-12-15T22:57:33.632897+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"55'26","reported_seq":145,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-12-15T22:57:35.519626+0000","last_change":"2023-12-15T22:57:32.649735+0000","last_active":"2023-12-15T22:57:35.519626+0000","last_peered":"2023-12-15T22:57:35.519626+0000","last_clean":"2023-12-15T22:57:35.519626+0000","last_became_active":"2023-12-15T22:56:39.560980+0000","last_became_peered":"2023-12-15T22:56:39.560980+0000","last_unstale":"2023-12-15T22:57:35.519626+0000","last_undegraded":"2023-12-15T22:57:35.519626+0000","last_fullsized":"2023-12-15T22:57:35.519626+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5747961,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":13740,"num_write":21,"num_write_kb":6063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":103,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-12-15T22:57:35.525302+0000","last_change":"2023-12-15T22:56:33.648541+0000","last_active":"2023-12-15T22:57:35.525302+0000","last_peered":"2023-12-15T22:57:35.525302+0000","last_clean":"2023-12-15T22:57:35.525302+0000","last_became_active":"2023-12-15T22:56:33.643681+0000","last_became_peered":"2023-12-15T22:56:33.643681+0000","last_unstale":"2023-12-15T22:57:35.525302+0000","last_undegraded":"2023-12-15T22:57:35.525302+0000","last_fullsized":"2023-12-15T22:57:35.525302+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"61'57","reported_seq":190,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-15T22:57:34.543535+0000","last_change":"2023-12-15T22:57:32.628785+0000","last_active":"2023-12-15T22:57:34.543535+0000","last_peered":"2023-12-15T22:57:34.543535+0000","last_clean":"2023-12-15T22:57:34.543535+0000","last_became_active":"2023-12-15T22:56:39.506510+0000","last_became_peered":"2023-12-15T22:56:39.506510+0000","last_unstale":"2023-12-15T22:57:34.543535+0000","last_undegraded":"2023-12-15T22:57:34.543535+0000","last_fullsized":"2023-12-15T22:57:34.543535+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8789995,"num_objects":10,"num_object_clones":5,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":59,"num_read_kb":42594,"num_write":44,"num_write_kb":13017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"50'13","reported_seq":125,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-15T22:57:33.636744+0000","last_change":"2023-12-15T22:57:32.749046+0000","last_active":"2023-12-15T22:57:33.636744+0000","last_peered":"2023-12-15T22:57:33.636744+0000","last_clean":"2023-12-15T22:57:33.636744+0000","last_became_active":"2023-12-15T22:56:39.511159+0000","last_became_peered":"2023-12-15T22:56:39.511159+0000","last_unstale":"2023-12-15T22:57:33.636744+0000","last_undegraded":"2023-12-15T22:57:33.636744+0000","last_fullsized":"2023-12-15T22:57:33.636744+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":5,"num_read_kb":2,"num_write":9,"num_write_kb":2133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":103,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-15T22:57:34.533926+0000","last_change":"2023-12-15T22:56:33.633890+0000","last_active":"2023-12-15T22:57:34.533926+0000","last_peered":"2023-12-15T22:57:34.533926+0000","last_clean":"2023-12-15T22:57:34.533926+0000","last_became_active":"2023-12-15T22:56:33.631134+0000","last_became_peered":"2023-12-15T22:56:33.631134+0000","last_unstale":"2023-12-15T22:57:34.533926+0000","last_undegraded":"2023-12-15T22:57:34.533926+0000","last_fullsized":"2023-12-15T22:57:34.533926+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"54'47","reported_seq":163,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-15T22:57:34.542803+0000","last_change":"2023-12-15T22:57:32.624144+0000","last_active":"2023-12-15T22:57:34.542803+0000","last_peered":"2023-12-15T22:57:34.542803+0000","last_clean":"2023-12-15T22:57:34.542803+0000","last_became_active":"2023-12-15T22:56:39.532730+0000","last_became_peered":"2023-12-15T22:56:39.532730+0000","last_unstale":"2023-12-15T22:57:34.542803+0000","last_undegraded":"2023-12-15T22:57:34.542803+0000","last_fullsized":"2023-12-15T22:57:34.542803+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3764735,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":35,"num_read_kb":28298,"num_write":27,"num_write_kb":8830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":97,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-12-15T22:57:35.522911+0000","last_change":"2023-12-15T22:56:33.655736+0000","last_active":"2023-12-15T22:57:35.522911+0000","last_peered":"2023-12-15T22:57:35.522911+0000","last_clean":"2023-12-15T22:57:35.522911+0000","last_became_active":"2023-12-15T22:56:33.652917+0000","last_became_peered":"2023-12-15T22:56:33.652917+0000","last_unstale":"2023-12-15T22:57:35.522911+0000","last_undegraded":"2023-12-15T22:57:35.522911+0000","last_fullsized":"2023-12-15T22:57:35.522911+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"67'43","reported_seq":175,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-12-15T22:57:35.517325+0000","last_change":"2023-12-15T22:57:32.648421+0000","last_active":"2023-12-15T22:57:35.517325+0000","last_peered":"2023-12-15T22:57:35.517325+0000","last_clean":"2023-12-15T22:57:35.517325+0000","last_became_active":"2023-12-15T22:56:39.557252+0000","last_became_peered":"2023-12-15T22:56:39.557252+0000","last_unstale":"2023-12-15T22:57:35.517325+0000","last_undegraded":"2023-12-15T22:57:35.517325+0000","last_fullsized":"2023-12-15T22:57:35.517325+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3579858,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":51,"num_read_kb":39064,"num_write":36,"num_write_kb":11126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":103,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-15T22:57:34.539537+0000","last_change":"2023-12-15T22:56:33.632348+0000","last_active":"2023-12-15T22:57:34.539537+0000","last_peered":"2023-12-15T22:57:34.539537+0000","last_clean":"2023-12-15T22:57:34.539537+0000","last_became_active":"2023-12-15T22:56:33.629022+0000","last_became_peered":"2023-12-15T22:56:33.629022+0000","last_unstale":"2023-12-15T22:57:34.539537+0000","last_undegraded":"2023-12-15T22:57:34.539537+0000","last_fullsized":"2023-12-15T22:57:34.539537+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"64'7","reported_seq":134,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-15T22:57:34.536133+0000","last_change":"2023-12-15T22:57:32.628314+0000","last_active":"2023-12-15T22:57:34.536133+0000","last_peered":"2023-12-15T22:57:34.536133+0000","last_clean":"2023-12-15T22:57:34.536133+0000","last_became_active":"2023-12-15T22:56:39.535145+0000","last_became_peered":"2023-12-15T22:56:39.535145+0000","last_unstale":"2023-12-15T22:57:34.536133+0000","last_undegraded":"2023-12-15T22:57:34.536133+0000","last_fullsized":"2023-12-15T22:57:34.536133+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2756348,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":7487,"num_write":8,"num_write_kb":2448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":97,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-15T22:57:33.633436+0000","last_change":"2023-12-15T22:56:33.627005+0000","last_active":"2023-12-15T22:57:33.633436+0000","last_peered":"2023-12-15T22:57:33.633436+0000","last_clean":"2023-12-15T22:57:33.633436+0000","last_became_active":"2023-12-15T22:56:33.622181+0000","last_became_peered":"2023-12-15T22:56:33.622181+0000","last_unstale":"2023-12-15T22:57:33.633436+0000","last_undegraded":"2023-12-15T22:57:33.633436+0000","last_fullsized":"2023-12-15T22:57:33.633436+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":100959154,"num_objects":108,"num_object_clones":57,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":643,"num_read_kb":411903,"num_write":582,"num_write_kb":166058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":734,"ondisk_log_size":734,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705729,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":79648,"kb_used_data":52824,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94292192,"statfs":{"total":96636764160,"available":96555204608,"internally_reserved":0,"allocated":54091776,"data_stored":58068410,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705728,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":72116,"kb_used_data":45292,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94299724,"statfs":{"total":96636764160,"available":96562917376,"internally_reserved":0,"allocated":46379008,"data_stored":50127435,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705728,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":76000,"kb_used_data":49176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94295840,"statfs":{"total":96636764160,"available":96558940160,"internally_reserved":0,"allocated":50356224,"data_stored":52580801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705729,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":85396,"kb_used_data":58572,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94286444,"statfs":{"total":96636764160,"available":96549318656,"internally_reserved":0,"allocated":59977728,"data_stored":64750816,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-15T22:57:36.612 INFO:tasks.rados.rados.0.smithi071.stdout:301: done (2 left) 2023-12-15T22:57:36.613 INFO:tasks.rados.rados.0.smithi071.stdout:302: write oid 31 current snap is 27 2023-12-15T22:57:36.613 INFO:tasks.rados.rados.0.smithi071.stdout:302: seq_num 171 ranges {473997=627345,1673362=441052,2529567=344430} 2023-12-15T22:57:36.618 INFO:tasks.rados.rados.0.smithi071.stdout:302: writing smithi07183189-31 from 473997 to 1101342 tid 1 2023-12-15T22:57:36.624 INFO:tasks.rados.rados.0.smithi071.stdout:302: writing smithi07183189-31 from 1673362 to 2114414 tid 2 2023-12-15T22:57:36.628 INFO:tasks.rados.rados.0.smithi071.stdout:302: writing smithi07183189-31 from 2529567 to 2873997 tid 3 2023-12-15T22:57:36.628 INFO:tasks.rados.rados.0.smithi071.stdout:303: read oid 39 snap -1 2023-12-15T22:57:36.629 INFO:tasks.rados.rados.0.smithi071.stdout:303: expect deleted 2023-12-15T22:57:36.629 INFO:tasks.rados.rados.0.smithi071.stdout:304: write (excl) oid 17 current snap is 27 2023-12-15T22:57:36.629 INFO:tasks.rados.rados.0.smithi071.stdout:304: seq_num 172 ranges {490804=702594,1615491=584710,2724284=166520} 2023-12-15T22:57:36.635 INFO:tasks.rados.rados.0.smithi071.stdout:304: writing smithi07183189-17 from 490804 to 1193398 tid 1 2023-12-15T22:57:36.641 INFO:tasks.rados.rados.0.smithi071.stdout:304: writing smithi07183189-17 from 1615491 to 2200201 tid 2 2023-12-15T22:57:36.645 INFO:tasks.rados.rados.0.smithi071.stdout:304: writing smithi07183189-17 from 2724284 to 2890804 tid 3 2023-12-15T22:57:36.645 INFO:tasks.rados.rados.0.smithi071.stdout:305: snap_create 2023-12-15T22:57:36.645 INFO:tasks.rados.rados.0.smithi071.stdout:300: expect (ObjNum 66 snap 3 seq_num 66) 2023-12-15T22:57:36.813 INFO:tasks.rados.rados.0.smithi071.stdout:304: finishing write tid 1 to smithi07183189-17 2023-12-15T22:57:36.813 INFO:tasks.rados.rados.0.smithi071.stdout:302: finishing write tid 1 to smithi07183189-31 2023-12-15T22:57:36.813 INFO:tasks.rados.rados.0.smithi071.stdout:304: finishing write tid 2 to smithi07183189-17 2023-12-15T22:57:36.813 INFO:tasks.rados.rados.0.smithi071.stdout:302: finishing write tid 2 to smithi07183189-31 2023-12-15T22:57:36.813 INFO:tasks.rados.rados.0.smithi071.stdout:304: finishing write tid 3 to smithi07183189-17 2023-12-15T22:57:36.813 INFO:tasks.rados.rados.0.smithi071.stdout:302: finishing write tid 3 to smithi07183189-31 2023-12-15T22:57:36.813 INFO:tasks.rados.rados.0.smithi071.stdout:304: finishing write tid 4 to smithi07183189-17 2023-12-15T22:57:36.814 INFO:tasks.rados.rados.0.smithi071.stdout:304: finishing write tid 5 to smithi07183189-17 2023-12-15T22:57:36.814 INFO:tasks.rados.rados.0.smithi071.stdout:304: oid 17 updating version 0 to 98 2023-12-15T22:57:36.814 INFO:tasks.rados.rados.0.smithi071.stdout:304: oid 17 updating version 98 to 100 2023-12-15T22:57:36.814 INFO:tasks.rados.rados.0.smithi071.stdout:304: oid 17 version 100 is already newer than 100 2023-12-15T22:57:36.814 INFO:tasks.rados.rados.0.smithi071.stdout:304: oid 17 version 100 is already newer than 97 2023-12-15T22:57:36.814 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 17 v 100 (ObjNum 172 snap 27 seq_num 172) dirty exists 2023-12-15T22:57:36.814 INFO:tasks.rados.rados.0.smithi071.stdout:304: left oid 17 (ObjNum 172 snap 27 seq_num 172) 2023-12-15T22:57:36.814 INFO:tasks.rados.rados.0.smithi071.stdout:302: finishing write tid 4 to smithi07183189-31 2023-12-15T22:57:36.814 INFO:tasks.rados.rados.0.smithi071.stdout:302: finishing write tid 5 to smithi07183189-31 2023-12-15T22:57:36.814 INFO:tasks.rados.rados.0.smithi071.stdout:302: oid 31 updating version 0 to 48 2023-12-15T22:57:36.814 INFO:tasks.rados.rados.0.smithi071.stdout:302: oid 31 version 48 is already newer than 47 2023-12-15T22:57:36.814 INFO:tasks.rados.rados.0.smithi071.stdout:302: oid 31 updating version 48 to 50 2023-12-15T22:57:36.814 INFO:tasks.rados.rados.0.smithi071.stdout:302: oid 31 version 50 is already newer than 49 2023-12-15T22:57:36.814 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 50 (ObjNum 171 snap 27 seq_num 171) dirty exists 2023-12-15T22:57:36.814 INFO:tasks.rados.rados.0.smithi071.stdout:302: left oid 31 (ObjNum 171 snap 27 seq_num 171) 2023-12-15T22:57:37.107 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:57:37.210 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:37.209+0000 7f4b1b06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:37.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:37.213+0000 7f4b1b06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:37.235 INFO:tasks.rados.rados.0.smithi071.stdout:298: done (5 left) 2023-12-15T22:57:37.235 INFO:tasks.rados.rados.0.smithi071.stdout:300: done (4 left) 2023-12-15T22:57:37.235 INFO:tasks.rados.rados.0.smithi071.stdout:302: done (3 left) 2023-12-15T22:57:37.235 INFO:tasks.rados.rados.0.smithi071.stdout:303: done (2 left) 2023-12-15T22:57:37.235 INFO:tasks.rados.rados.0.smithi071.stdout:304: done (1 left) 2023-12-15T22:57:37.235 INFO:tasks.rados.rados.0.smithi071.stdout:305: done (0 left) 2023-12-15T22:57:37.235 INFO:tasks.rados.rados.0.smithi071.stdout:306: read oid 16 snap -1 2023-12-15T22:57:37.235 INFO:tasks.rados.rados.0.smithi071.stdout:306: expect deleted 2023-12-15T22:57:37.235 INFO:tasks.rados.rados.0.smithi071.stdout:307: delete oid 37 current snap is 28 2023-12-15T22:57:37.344 INFO:tasks.rados.rados.0.smithi071.stdout:306: done (1 left) 2023-12-15T22:57:37.344 INFO:tasks.rados.rados.0.smithi071.stdout:307: done (0 left) 2023-12-15T22:57:37.344 INFO:tasks.rados.rados.0.smithi071.stdout:308: snap_remove snap 27 2023-12-15T22:57:38.238 INFO:tasks.rados.rados.0.smithi071.stdout:308: done (0 left) 2023-12-15T22:57:38.239 INFO:tasks.rados.rados.0.smithi071.stdout:309: snap_create 2023-12-15T22:57:39.243 INFO:tasks.rados.rados.0.smithi071.stdout:309: done (0 left) 2023-12-15T22:57:39.243 INFO:tasks.rados.rados.0.smithi071.stdout:310: write oid 32 current snap is 29 2023-12-15T22:57:39.243 INFO:tasks.rados.rados.0.smithi071.stdout:310: seq_num 174 ranges {524418=453092,1732455=588379,2779130=145288} 2023-12-15T22:57:39.248 INFO:tasks.rados.rados.0.smithi071.stdout:310: writing smithi07183189-32 from 524418 to 977510 tid 1 2023-12-15T22:57:39.257 INFO:tasks.rados.rados.0.smithi071.stdout:310: writing smithi07183189-32 from 1732455 to 2320834 tid 2 2023-12-15T22:57:39.262 INFO:tasks.rados.rados.0.smithi071.stdout:310: writing smithi07183189-32 from 2779130 to 2924418 tid 3 2023-12-15T22:57:39.262 INFO:tasks.rados.rados.0.smithi071.stdout:311: read oid 36 snap 28 2023-12-15T22:57:39.262 INFO:tasks.rados.rados.0.smithi071.stdout:311: expect deleted 2023-12-15T22:57:39.262 INFO:tasks.rados.rados.0.smithi071.stdout:312: snap_create 2023-12-15T22:57:39.291 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:57:39.340 INFO:tasks.rados.rados.0.smithi071.stdout:310: finishing write tid 1 to smithi07183189-32 2023-12-15T22:57:39.343 INFO:tasks.rados.rados.0.smithi071.stdout:310: finishing write tid 2 to smithi07183189-32 2023-12-15T22:57:39.353 INFO:tasks.rados.rados.0.smithi071.stdout:310: finishing write tid 3 to smithi07183189-32 2023-12-15T22:57:39.361 INFO:tasks.rados.rados.0.smithi071.stdout:310: finishing write tid 4 to smithi07183189-32 2023-12-15T22:57:39.362 INFO:tasks.rados.rados.0.smithi071.stdout:310: finishing write tid 5 to smithi07183189-32 2023-12-15T22:57:39.363 INFO:tasks.rados.rados.0.smithi071.stdout:310: oid 32 updating version 0 to 24 2023-12-15T22:57:39.363 INFO:tasks.rados.rados.0.smithi071.stdout:310: oid 32 version 24 is already newer than 21 2023-12-15T22:57:39.363 INFO:tasks.rados.rados.0.smithi071.stdout:310: oid 32 version 24 is already newer than 22 2023-12-15T22:57:39.363 INFO:tasks.rados.rados.0.smithi071.stdout:310: oid 32 version 24 is already newer than 23 2023-12-15T22:57:39.363 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 24 (ObjNum 174 snap 29 seq_num 174) dirty exists 2023-12-15T22:57:39.363 INFO:tasks.rados.rados.0.smithi071.stdout:310: left oid 32 (ObjNum 174 snap 29 seq_num 174) 2023-12-15T22:57:39.403 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:39.402+0000 7f5bf5a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:39.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:39.406+0000 7f5bf5a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:40.248 INFO:tasks.rados.rados.0.smithi071.stdout:310: done (2 left) 2023-12-15T22:57:40.248 INFO:tasks.rados.rados.0.smithi071.stdout:311: done (1 left) 2023-12-15T22:57:40.248 INFO:tasks.rados.rados.0.smithi071.stdout:312: done (0 left) 2023-12-15T22:57:40.248 INFO:tasks.rados.rados.0.smithi071.stdout:313: read oid 23 snap -1 2023-12-15T22:57:40.248 INFO:tasks.rados.rados.0.smithi071.stdout:313: expect deleted 2023-12-15T22:57:40.248 INFO:tasks.rados.rados.0.smithi071.stdout:314: snap_create 2023-12-15T22:57:40.746 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T22:57:40.746 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T22:57:40.746 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T22:57:40.868 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:40.863+0000 7f8cad406640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:40.870 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:40.870+0000 7f8cad406640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:41.046 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:57:41.046 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-12-15T22:57:41.057 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2023-12-15T22:57:40.525459+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":104440326,"num_objects":111,"num_object_clones":57,"num_object_copies":333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":754,"num_read_kb":419590,"num_write":724,"num_write_kb":169374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":843,"ondisk_log_size":843,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":310016,"kb_used_data":202720,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377177344,"statfs":{"total":386547056640,"available":386229600256,"internally_reserved":0,"allocated":207585280,"data_stored":219579120,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2890804,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":7596,"num_write":5,"num_write_kb":1422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.915302"},"pg_stats":[{"pgid":"3.f","version":"66'18","reported_seq":143,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-12-15T22:57:35.521892+0000","last_change":"2023-12-15T22:57:33.642213+0000","last_active":"2023-12-15T22:57:35.521892+0000","last_peered":"2023-12-15T22:57:35.521892+0000","last_clean":"2023-12-15T22:57:35.521892+0000","last_became_active":"2023-12-15T22:56:39.503797+0000","last_became_peered":"2023-12-15T22:56:39.503797+0000","last_unstale":"2023-12-15T22:57:35.521892+0000","last_undegraded":"2023-12-15T22:57:35.521892+0000","last_fullsized":"2023-12-15T22:57:35.521892+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2016807,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":25,"num_read_kb":16106,"num_write":12,"num_write_kb":3599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":131,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-12-15T22:57:37.636774+0000","last_change":"2023-12-15T22:57:36.701441+0000","last_active":"2023-12-15T22:57:37.636774+0000","last_peered":"2023-12-15T22:57:37.636774+0000","last_clean":"2023-12-15T22:57:37.636774+0000","last_became_active":"2023-12-15T22:56:39.552856+0000","last_became_peered":"2023-12-15T22:56:39.552856+0000","last_unstale":"2023-12-15T22:57:37.636774+0000","last_undegraded":"2023-12-15T22:57:37.636774+0000","last_fullsized":"2023-12-15T22:57:37.636774+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"70'100","reported_seq":244,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-12-15T22:57:39.296528+0000","last_change":"2023-12-15T22:57:39.306814+0000","last_active":"2023-12-15T22:57:39.296528+0000","last_peered":"2023-12-15T22:57:39.296528+0000","last_clean":"2023-12-15T22:57:39.296528+0000","last_became_active":"2023-12-15T22:56:39.495490+0000","last_became_peered":"2023-12-15T22:56:39.495490+0000","last_unstale":"2023-12-15T22:57:39.296528+0000","last_undegraded":"2023-12-15T22:57:39.296528+0000","last_fullsized":"2023-12-15T22:57:39.296528+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11966551,"num_objects":14,"num_object_clones":7,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":96,"num_read_kb":65687,"num_write":80,"num_write_kb":22334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"67'61","reported_seq":193,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-12-15T22:57:35.534808+0000","last_change":"2023-12-15T22:57:33.653945+0000","last_active":"2023-12-15T22:57:35.534808+0000","last_peered":"2023-12-15T22:57:35.534808+0000","last_clean":"2023-12-15T22:57:35.534808+0000","last_became_active":"2023-12-15T22:56:39.579234+0000","last_became_peered":"2023-12-15T22:56:39.579234+0000","last_unstale":"2023-12-15T22:57:35.534808+0000","last_undegraded":"2023-12-15T22:57:35.534808+0000","last_fullsized":"2023-12-15T22:57:35.534808+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9025326,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":57,"num_read_kb":37632,"num_write":53,"num_write_kb":14095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"64'23","reported_seq":143,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-12-15T22:57:35.524170+0000","last_change":"2023-12-15T22:57:32.636891+0000","last_active":"2023-12-15T22:57:35.524170+0000","last_peered":"2023-12-15T22:57:35.524170+0000","last_clean":"2023-12-15T22:57:35.524170+0000","last_became_active":"2023-12-15T22:56:39.572176+0000","last_became_peered":"2023-12-15T22:56:39.572176+0000","last_unstale":"2023-12-15T22:57:35.524170+0000","last_undegraded":"2023-12-15T22:57:35.524170+0000","last_fullsized":"2023-12-15T22:57:35.524170+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6050487,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":3686,"num_write":21,"num_write_kb":6135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"66'34","reported_seq":158,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-12-15T22:57:35.522732+0000","last_change":"2023-12-15T22:57:33.642867+0000","last_active":"2023-12-15T22:57:35.522732+0000","last_peered":"2023-12-15T22:57:35.522732+0000","last_clean":"2023-12-15T22:57:35.522732+0000","last_became_active":"2023-12-15T22:56:39.516361+0000","last_became_peered":"2023-12-15T22:56:39.516361+0000","last_unstale":"2023-12-15T22:57:35.522732+0000","last_undegraded":"2023-12-15T22:57:35.522732+0000","last_fullsized":"2023-12-15T22:57:35.522732+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4924418,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":19607,"num_write":31,"num_write_kb":8847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"67'154","reported_seq":270,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-12-15T22:57:35.519658+0000","last_change":"2023-12-15T22:57:33.642515+0000","last_active":"2023-12-15T22:57:35.519658+0000","last_peered":"2023-12-15T22:57:35.519658+0000","last_clean":"2023-12-15T22:57:35.519658+0000","last_became_active":"2023-12-15T22:56:39.519748+0000","last_became_peered":"2023-12-15T22:56:39.519748+0000","last_unstale":"2023-12-15T22:57:35.519658+0000","last_undegraded":"2023-12-15T22:57:35.519658+0000","last_fullsized":"2023-12-15T22:57:35.519658+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20017104,"num_objects":24,"num_object_clones":14,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":119,"num_read_kb":67758,"num_write":110,"num_write_kb":31583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"69'44","reported_seq":183,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-12-15T22:57:39.302365+0000","last_change":"2023-12-15T22:57:39.309894+0000","last_active":"2023-12-15T22:57:39.302365+0000","last_peered":"2023-12-15T22:57:39.302365+0000","last_clean":"2023-12-15T22:57:39.302365+0000","last_became_active":"2023-12-15T22:56:39.543179+0000","last_became_peered":"2023-12-15T22:56:39.543179+0000","last_unstale":"2023-12-15T22:57:39.302365+0000","last_undegraded":"2023-12-15T22:57:39.302365+0000","last_fullsized":"2023-12-15T22:57:39.302365+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8857223,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":33357,"num_write":40,"num_write_kb":10468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":226,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-12-15T22:57:35.526074+0000","last_change":"2023-12-15T22:56:32.607017+0000","last_active":"2023-12-15T22:57:35.526074+0000","last_peered":"2023-12-15T22:57:35.526074+0000","last_clean":"2023-12-15T22:57:35.526074+0000","last_became_active":"2023-12-15T22:56:32.602384+0000","last_became_peered":"2023-12-15T22:56:32.602384+0000","last_unstale":"2023-12-15T22:57:35.526074+0000","last_undegraded":"2023-12-15T22:57:35.526074+0000","last_fullsized":"2023-12-15T22:57:35.526074+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_clean_scrub_stamp":"2023-12-15T22:56:31.525572+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":107,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-12-15T22:57:39.307238+0000","last_change":"2023-12-15T22:56:33.627481+0000","last_active":"2023-12-15T22:57:39.307238+0000","last_peered":"2023-12-15T22:57:39.307238+0000","last_clean":"2023-12-15T22:57:39.307238+0000","last_became_active":"2023-12-15T22:56:33.624689+0000","last_became_peered":"2023-12-15T22:56:33.624689+0000","last_unstale":"2023-12-15T22:57:39.307238+0000","last_undegraded":"2023-12-15T22:57:39.307238+0000","last_fullsized":"2023-12-15T22:57:39.307238+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"69'64","reported_seq":195,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-12-15T22:57:39.288663+0000","last_change":"2023-12-15T22:57:39.296761+0000","last_active":"2023-12-15T22:57:39.288663+0000","last_peered":"2023-12-15T22:57:39.288663+0000","last_clean":"2023-12-15T22:57:39.288663+0000","last_became_active":"2023-12-15T22:56:39.529359+0000","last_became_peered":"2023-12-15T22:56:39.529359+0000","last_unstale":"2023-12-15T22:57:39.288663+0000","last_undegraded":"2023-12-15T22:57:39.288663+0000","last_fullsized":"2023-12-15T22:57:39.288663+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7176589,"num_objects":8,"num_object_clones":3,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":36,"num_read_kb":10704,"num_write":56,"num_write_kb":16150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":103,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-12-15T22:57:35.524456+0000","last_change":"2023-12-15T22:56:33.655797+0000","last_active":"2023-12-15T22:57:35.524456+0000","last_peered":"2023-12-15T22:57:35.524456+0000","last_clean":"2023-12-15T22:57:35.524456+0000","last_became_active":"2023-12-15T22:56:33.652885+0000","last_became_peered":"2023-12-15T22:56:33.652885+0000","last_unstale":"2023-12-15T22:57:35.524456+0000","last_undegraded":"2023-12-15T22:57:35.524456+0000","last_fullsized":"2023-12-15T22:57:35.524456+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"62'53","reported_seq":177,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-12-15T22:57:35.522195+0000","last_change":"2023-12-15T22:57:32.640556+0000","last_active":"2023-12-15T22:57:35.522195+0000","last_peered":"2023-12-15T22:57:35.522195+0000","last_clean":"2023-12-15T22:57:35.522195+0000","last_became_active":"2023-12-15T22:56:39.543509+0000","last_became_peered":"2023-12-15T22:56:39.543509+0000","last_unstale":"2023-12-15T22:57:35.522195+0000","last_undegraded":"2023-12-15T22:57:35.522195+0000","last_fullsized":"2023-12-15T22:57:35.522195+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9176556,"num_objects":9,"num_object_clones":4,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":52,"num_read_kb":33777,"num_write":39,"num_write_kb":10652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":107,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-12-15T22:57:39.298962+0000","last_change":"2023-12-15T22:56:33.600485+0000","last_active":"2023-12-15T22:57:39.298962+0000","last_peered":"2023-12-15T22:57:39.298962+0000","last_clean":"2023-12-15T22:57:39.298962+0000","last_became_active":"2023-12-15T22:56:33.597320+0000","last_became_peered":"2023-12-15T22:56:33.597320+0000","last_unstale":"2023-12-15T22:57:39.298962+0000","last_undegraded":"2023-12-15T22:57:39.298962+0000","last_fullsized":"2023-12-15T22:57:39.298962+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"55'26","reported_seq":145,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-12-15T22:57:35.519626+0000","last_change":"2023-12-15T22:57:32.649735+0000","last_active":"2023-12-15T22:57:35.519626+0000","last_peered":"2023-12-15T22:57:35.519626+0000","last_clean":"2023-12-15T22:57:35.519626+0000","last_became_active":"2023-12-15T22:56:39.560980+0000","last_became_peered":"2023-12-15T22:56:39.560980+0000","last_unstale":"2023-12-15T22:57:35.519626+0000","last_undegraded":"2023-12-15T22:57:35.519626+0000","last_fullsized":"2023-12-15T22:57:35.519626+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5747961,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":13740,"num_write":21,"num_write_kb":6063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":103,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-12-15T22:57:35.525302+0000","last_change":"2023-12-15T22:56:33.648541+0000","last_active":"2023-12-15T22:57:35.525302+0000","last_peered":"2023-12-15T22:57:35.525302+0000","last_clean":"2023-12-15T22:57:35.525302+0000","last_became_active":"2023-12-15T22:56:33.643681+0000","last_became_peered":"2023-12-15T22:56:33.643681+0000","last_unstale":"2023-12-15T22:57:35.525302+0000","last_undegraded":"2023-12-15T22:57:35.525302+0000","last_fullsized":"2023-12-15T22:57:35.525302+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"61'57","reported_seq":198,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-12-15T22:57:37.634001+0000","last_change":"2023-12-15T22:57:36.684866+0000","last_active":"2023-12-15T22:57:37.634001+0000","last_peered":"2023-12-15T22:57:37.634001+0000","last_clean":"2023-12-15T22:57:37.634001+0000","last_became_active":"2023-12-15T22:56:39.506510+0000","last_became_peered":"2023-12-15T22:56:39.506510+0000","last_unstale":"2023-12-15T22:57:37.634001+0000","last_undegraded":"2023-12-15T22:57:37.634001+0000","last_fullsized":"2023-12-15T22:57:37.634001+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8789995,"num_objects":10,"num_object_clones":5,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":59,"num_read_kb":42594,"num_write":44,"num_write_kb":13017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"50'13","reported_seq":138,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-12-15T22:57:39.298000+0000","last_change":"2023-12-15T22:57:39.307018+0000","last_active":"2023-12-15T22:57:39.298000+0000","last_peered":"2023-12-15T22:57:39.298000+0000","last_clean":"2023-12-15T22:57:39.298000+0000","last_became_active":"2023-12-15T22:56:39.511159+0000","last_became_peered":"2023-12-15T22:56:39.511159+0000","last_unstale":"2023-12-15T22:57:39.298000+0000","last_undegraded":"2023-12-15T22:57:39.298000+0000","last_fullsized":"2023-12-15T22:57:39.298000+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":5,"num_read_kb":2,"num_write":9,"num_write_kb":2133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":109,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-12-15T22:57:37.637475+0000","last_change":"2023-12-15T22:56:33.633890+0000","last_active":"2023-12-15T22:57:37.637475+0000","last_peered":"2023-12-15T22:57:37.637475+0000","last_clean":"2023-12-15T22:57:37.637475+0000","last_became_active":"2023-12-15T22:56:33.631134+0000","last_became_peered":"2023-12-15T22:56:33.631134+0000","last_unstale":"2023-12-15T22:57:37.637475+0000","last_undegraded":"2023-12-15T22:57:37.637475+0000","last_fullsized":"2023-12-15T22:57:37.637475+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"54'47","reported_seq":171,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-12-15T22:57:37.635534+0000","last_change":"2023-12-15T22:57:36.701916+0000","last_active":"2023-12-15T22:57:37.635534+0000","last_peered":"2023-12-15T22:57:37.635534+0000","last_clean":"2023-12-15T22:57:37.635534+0000","last_became_active":"2023-12-15T22:56:39.532730+0000","last_became_peered":"2023-12-15T22:56:39.532730+0000","last_unstale":"2023-12-15T22:57:37.635534+0000","last_undegraded":"2023-12-15T22:57:37.635534+0000","last_fullsized":"2023-12-15T22:57:37.635534+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3764735,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":35,"num_read_kb":28298,"num_write":27,"num_write_kb":8830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":97,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-12-15T22:57:35.522911+0000","last_change":"2023-12-15T22:56:33.655736+0000","last_active":"2023-12-15T22:57:35.522911+0000","last_peered":"2023-12-15T22:57:35.522911+0000","last_clean":"2023-12-15T22:57:35.522911+0000","last_became_active":"2023-12-15T22:56:33.652917+0000","last_became_peered":"2023-12-15T22:56:33.652917+0000","last_unstale":"2023-12-15T22:57:35.522911+0000","last_undegraded":"2023-12-15T22:57:35.522911+0000","last_fullsized":"2023-12-15T22:57:35.522911+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"67'43","reported_seq":175,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-12-15T22:57:35.517325+0000","last_change":"2023-12-15T22:57:32.648421+0000","last_active":"2023-12-15T22:57:35.517325+0000","last_peered":"2023-12-15T22:57:35.517325+0000","last_clean":"2023-12-15T22:57:35.517325+0000","last_became_active":"2023-12-15T22:56:39.557252+0000","last_became_peered":"2023-12-15T22:56:39.557252+0000","last_unstale":"2023-12-15T22:57:35.517325+0000","last_undegraded":"2023-12-15T22:57:35.517325+0000","last_fullsized":"2023-12-15T22:57:35.517325+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3579858,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":51,"num_read_kb":39064,"num_write":36,"num_write_kb":11126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":109,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-12-15T22:57:37.621195+0000","last_change":"2023-12-15T22:56:33.632348+0000","last_active":"2023-12-15T22:57:37.621195+0000","last_peered":"2023-12-15T22:57:37.621195+0000","last_clean":"2023-12-15T22:57:37.621195+0000","last_became_active":"2023-12-15T22:56:33.629022+0000","last_became_peered":"2023-12-15T22:56:33.629022+0000","last_unstale":"2023-12-15T22:57:37.621195+0000","last_undegraded":"2023-12-15T22:57:37.621195+0000","last_fullsized":"2023-12-15T22:57:37.621195+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"70'9","reported_seq":142,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-12-15T22:57:37.644015+0000","last_change":"2023-12-15T22:57:37.644015+0000","last_active":"2023-12-15T22:57:37.644015+0000","last_peered":"2023-12-15T22:57:37.644015+0000","last_clean":"2023-12-15T22:57:37.644015+0000","last_became_active":"2023-12-15T22:56:39.535145+0000","last_became_peered":"2023-12-15T22:56:39.535145+0000","last_unstale":"2023-12-15T22:57:37.644015+0000","last_undegraded":"2023-12-15T22:57:37.644015+0000","last_fullsized":"2023-12-15T22:57:37.644015+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2756348,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":7487,"num_write":8,"num_write_kb":2448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":107,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-12-15T22:57:39.290096+0000","last_change":"2023-12-15T22:56:33.627005+0000","last_active":"2023-12-15T22:57:39.290096+0000","last_peered":"2023-12-15T22:57:39.290096+0000","last_clean":"2023-12-15T22:57:39.290096+0000","last_became_active":"2023-12-15T22:56:33.622181+0000","last_became_peered":"2023-12-15T22:56:33.622181+0000","last_unstale":"2023-12-15T22:57:39.290096+0000","last_undegraded":"2023-12-15T22:57:39.290096+0000","last_fullsized":"2023-12-15T22:57:39.290096+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":103849958,"num_objects":109,"num_object_clones":57,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":651,"num_read_kb":419499,"num_write":587,"num_write_kb":167480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":746,"ondisk_log_size":746,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705729,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":79648,"kb_used_data":52824,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94292192,"statfs":{"total":96636764160,"available":96555204608,"internally_reserved":0,"allocated":54091776,"data_stored":58068410,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705733,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70692,"kb_used_data":43868,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94301148,"statfs":{"total":96636764160,"available":96564375552,"internally_reserved":0,"allocated":44920832,"data_stored":46711697,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705733,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":74280,"kb_used_data":47456,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94297560,"statfs":{"total":96636764160,"available":96560701440,"internally_reserved":0,"allocated":48594944,"data_stored":50048197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705729,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":85396,"kb_used_data":58572,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94286444,"statfs":{"total":96636764160,"available":96549318656,"internally_reserved":0,"allocated":59977728,"data_stored":64750816,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-15T22:57:41.249 INFO:tasks.rados.rados.0.smithi071.stdout:313: done (1 left) 2023-12-15T22:57:41.249 INFO:tasks.rados.rados.0.smithi071.stdout:314: done (0 left) 2023-12-15T22:57:41.249 INFO:tasks.rados.rados.0.smithi071.stdout:315: snap_remove snap 30 2023-12-15T22:57:41.474 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:57:41.579 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:41.577+0000 7f02ea86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:41.582 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:41.582+0000 7f02ea86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:42.236 INFO:tasks.rados.rados.0.smithi071.stdout:315: done (0 left) 2023-12-15T22:57:42.237 INFO:tasks.rados.rados.0.smithi071.stdout:316: snap_create 2023-12-15T22:57:43.238 INFO:tasks.rados.rados.0.smithi071.stdout:316: done (0 left) 2023-12-15T22:57:43.238 INFO:tasks.rados.rados.0.smithi071.stdout:317: read oid 34 snap -1 2023-12-15T22:57:43.238 INFO:tasks.rados.rados.0.smithi071.stdout:317: expect (ObjNum 164 snap 24 seq_num 164) 2023-12-15T22:57:43.238 INFO:tasks.rados.rados.0.smithi071.stdout:318: delete oid 2 current snap is 32 2023-12-15T22:57:43.286 INFO:tasks.rados.rados.0.smithi071.stdout:317: expect (ObjNum 164 snap 24 seq_num 164) 2023-12-15T22:57:43.442 INFO:tasks.rados.rados.0.smithi071.stdout:317: done (1 left) 2023-12-15T22:57:43.442 INFO:tasks.rados.rados.0.smithi071.stdout:318: done (0 left) 2023-12-15T22:57:43.443 INFO:tasks.rados.rados.0.smithi071.stdout:319: snap_create 2023-12-15T22:57:43.653 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:57:43.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:43.754+0000 7f1683b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:43.759 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:43.759+0000 7f1683b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:44.257 INFO:tasks.rados.rados.0.smithi071.stdout:319: done (0 left) 2023-12-15T22:57:44.257 INFO:tasks.rados.rados.0.smithi071.stdout:320: read oid 2 snap -1 2023-12-15T22:57:44.257 INFO:tasks.rados.rados.0.smithi071.stdout:320: expect deleted 2023-12-15T22:57:44.257 INFO:tasks.rados.rados.0.smithi071.stdout:321: write (excl) oid 16 current snap is 33 2023-12-15T22:57:44.257 INFO:tasks.rados.rados.0.smithi071.stdout:321: seq_num 176 ranges {558032=719943,1849419=592048} 2023-12-15T22:57:44.263 INFO:tasks.rados.rados.0.smithi071.stdout:321: writing smithi07183189-16 from 558032 to 1277975 tid 1 2023-12-15T22:57:44.271 INFO:tasks.rados.rados.0.smithi071.stdout:321: writing smithi07183189-16 from 1849419 to 2441467 tid 2 2023-12-15T22:57:44.271 INFO:tasks.rados.rados.0.smithi071.stdout:322: write oid 13 current snap is 33 2023-12-15T22:57:44.271 INFO:tasks.rados.rados.0.smithi071.stdout:322: seq_num 177 ranges {574839=795192,1907901=452059} 2023-12-15T22:57:44.280 INFO:tasks.rados.rados.0.smithi071.stdout:322: writing smithi07183189-13 from 574839 to 1370031 tid 1 2023-12-15T22:57:44.287 INFO:tasks.rados.rados.0.smithi071.stdout:322: writing smithi07183189-13 from 1907901 to 2359960 tid 2 2023-12-15T22:57:44.287 INFO:tasks.rados.rados.0.smithi071.stdout:323: delete oid 22 current snap is 33 2023-12-15T22:57:44.361 INFO:tasks.rados.rados.0.smithi071.stdout:320: done (3 left) 2023-12-15T22:57:44.361 INFO:tasks.rados.rados.0.smithi071.stdout:323: done (2 left) 2023-12-15T22:57:44.361 INFO:tasks.rados.rados.0.smithi071.stdout:324: delete oid 33 current snap is 33 2023-12-15T22:57:44.362 INFO:tasks.rados.rados.0.smithi071.stdout:321: finishing write tid 1 to smithi07183189-16 2023-12-15T22:57:44.362 INFO:tasks.rados.rados.0.smithi071.stdout:322: finishing write tid 1 to smithi07183189-13 2023-12-15T22:57:44.387 INFO:tasks.rados.rados.0.smithi071.stdout:321: finishing write tid 2 to smithi07183189-16 2023-12-15T22:57:44.396 INFO:tasks.rados.rados.0.smithi071.stdout:324: done (2 left) 2023-12-15T22:57:44.396 INFO:tasks.rados.rados.0.smithi071.stdout:325: delete oid 28 current snap is 33 2023-12-15T22:57:44.397 INFO:tasks.rados.rados.0.smithi071.stdout:322: finishing write tid 2 to smithi07183189-13 2023-12-15T22:57:44.397 INFO:tasks.rados.rados.0.smithi071.stdout:321: finishing write tid 3 to smithi07183189-16 2023-12-15T22:57:44.399 INFO:tasks.rados.rados.0.smithi071.stdout:321: finishing write tid 4 to smithi07183189-16 2023-12-15T22:57:44.399 INFO:tasks.rados.rados.0.smithi071.stdout:321: oid 16 updating version 0 to 103 2023-12-15T22:57:44.399 INFO:tasks.rados.rados.0.smithi071.stdout:321: oid 16 version 103 is already newer than 103 2023-12-15T22:57:44.399 INFO:tasks.rados.rados.0.smithi071.stdout:321: oid 16 version 103 is already newer than 101 2023-12-15T22:57:44.399 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 16 v 103 (ObjNum 176 snap 33 seq_num 176) dirty exists 2023-12-15T22:57:44.399 INFO:tasks.rados.rados.0.smithi071.stdout:321: left oid 16 (ObjNum 176 snap 33 seq_num 176) 2023-12-15T22:57:44.419 INFO:tasks.rados.rados.0.smithi071.stdout:322: finishing write tid 3 to smithi07183189-13 2023-12-15T22:57:44.421 INFO:tasks.rados.rados.0.smithi071.stdout:322: finishing write tid 4 to smithi07183189-13 2023-12-15T22:57:44.421 INFO:tasks.rados.rados.0.smithi071.stdout:322: oid 13 updating version 0 to 56 2023-12-15T22:57:44.421 INFO:tasks.rados.rados.0.smithi071.stdout:322: oid 13 version 56 is already newer than 55 2023-12-15T22:57:44.421 INFO:tasks.rados.rados.0.smithi071.stdout:322: oid 13 version 56 is already newer than 54 2023-12-15T22:57:44.421 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 56 (ObjNum 177 snap 33 seq_num 177) dirty exists 2023-12-15T22:57:44.422 INFO:tasks.rados.rados.0.smithi071.stdout:322: left oid 13 (ObjNum 177 snap 33 seq_num 177) 2023-12-15T22:57:44.422 INFO:tasks.rados.rados.0.smithi071.stdout:321: done (2 left) 2023-12-15T22:57:44.422 INFO:tasks.rados.rados.0.smithi071.stdout:322: done (1 left) 2023-12-15T22:57:44.422 INFO:tasks.rados.rados.0.smithi071.stdout:325: done (0 left) 2023-12-15T22:57:44.422 INFO:tasks.rados.rados.0.smithi071.stdout:326: write oid 26 current snap is 33 2023-12-15T22:57:44.422 INFO:tasks.rados.rados.0.smithi071.stdout:326: seq_num 181 ranges {642067=696188,2025476=459397,2971091=70976} 2023-12-15T22:57:44.428 INFO:tasks.rados.rados.0.smithi071.stdout:326: writing smithi07183189-26 from 642067 to 1338255 tid 1 2023-12-15T22:57:44.434 INFO:tasks.rados.rados.0.smithi071.stdout:326: writing smithi07183189-26 from 2025476 to 2484873 tid 2 2023-12-15T22:57:44.437 INFO:tasks.rados.rados.0.smithi071.stdout:326: writing smithi07183189-26 from 2971091 to 3042067 tid 3 2023-12-15T22:57:44.437 INFO:tasks.rados.rados.0.smithi071.stdout:327: snap_remove snap 29 2023-12-15T22:57:45.262 INFO:tasks.rados.rados.0.smithi071.stdout:327: done (1 left) 2023-12-15T22:57:45.263 INFO:tasks.rados.rados.0.smithi071.stdout:328: snap_create 2023-12-15T22:57:45.263 INFO:tasks.rados.rados.0.smithi071.stdout:326: finishing write tid 1 to smithi07183189-26 2023-12-15T22:57:45.263 INFO:tasks.rados.rados.0.smithi071.stdout:326: finishing write tid 2 to smithi07183189-26 2023-12-15T22:57:45.263 INFO:tasks.rados.rados.0.smithi071.stdout:326: finishing write tid 3 to smithi07183189-26 2023-12-15T22:57:45.263 INFO:tasks.rados.rados.0.smithi071.stdout:326: finishing write tid 4 to smithi07183189-26 2023-12-15T22:57:45.263 INFO:tasks.rados.rados.0.smithi071.stdout:326: finishing write tid 5 to smithi07183189-26 2023-12-15T22:57:45.263 INFO:tasks.rados.rados.0.smithi071.stdout:326: oid 26 updating version 0 to 53 2023-12-15T22:57:45.263 INFO:tasks.rados.rados.0.smithi071.stdout:326: oid 26 updating version 53 to 54 2023-12-15T22:57:45.263 INFO:tasks.rados.rados.0.smithi071.stdout:326: oid 26 version 54 is already newer than 51 2023-12-15T22:57:45.263 INFO:tasks.rados.rados.0.smithi071.stdout:326: oid 26 version 54 is already newer than 53 2023-12-15T22:57:45.263 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 54 (ObjNum 181 snap 33 seq_num 181) dirty exists 2023-12-15T22:57:45.263 INFO:tasks.rados.rados.0.smithi071.stdout:326: left oid 26 (ObjNum 181 snap 33 seq_num 181) 2023-12-15T22:57:45.827 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:57:45.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:45.941+0000 7fb0ec79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:45.946 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:45.945+0000 7fb0ec79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:46.060 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T22:57:46.060 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T22:57:46.060 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T22:57:46.169 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:46.165+0000 7fa70ba6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:46.171 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:46.171+0000 7fa70ba6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:46.265 INFO:tasks.rados.rados.0.smithi071.stdout:326: done (1 left) 2023-12-15T22:57:46.266 INFO:tasks.rados.rados.0.smithi071.stdout:328: done (0 left) 2023-12-15T22:57:46.266 INFO:tasks.rados.rados.0.smithi071.stdout:329: write (excl) oid 29 current snap is 34 2023-12-15T22:57:46.266 INFO:tasks.rados.rados.0.smithi071.stdout:329: seq_num 182 ranges {658874=771437,2083958=603055} 2023-12-15T22:57:46.273 INFO:tasks.rados.rados.0.smithi071.stdout:329: writing smithi07183189-29 from 658874 to 1430311 tid 1 2023-12-15T22:57:46.281 INFO:tasks.rados.rados.0.smithi071.stdout:329: writing smithi07183189-29 from 2083958 to 2687013 tid 2 2023-12-15T22:57:46.281 INFO:tasks.rados.rados.0.smithi071.stdout:330: snap_remove snap 25 2023-12-15T22:57:46.359 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:57:46.359 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-12-15T22:57:46.371 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":111,"stamp":"2023-12-15T22:57:46.263599+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":101751305,"num_objects":121,"num_object_clones":63,"num_object_copies":363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":768,"num_read_kb":426171,"num_write":747,"num_write_kb":175616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":884,"ondisk_log_size":884,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":317544,"kb_used_data":206088,"kb_used_omap":0,"kb_used_meta":111424,"kb_avail":377169816,"statfs":{"total":386547056640,"available":386221891584,"internally_reserved":0,"allocated":211034112,"data_stored":218943624,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":114098176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2201651,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":12,"num_read_kb":6579,"num_write":13,"num_write_kb":3701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.028548"},"pg_stats":[{"pgid":"3.f","version":"73'24","reported_seq":165,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-12-15T22:57:45.275060+0000","last_change":"2023-12-15T22:57:45.280380+0000","last_active":"2023-12-15T22:57:45.275060+0000","last_peered":"2023-12-15T22:57:45.275060+0000","last_clean":"2023-12-15T22:57:45.275060+0000","last_became_active":"2023-12-15T22:56:39.503797+0000","last_became_peered":"2023-12-15T22:56:39.503797+0000","last_unstale":"2023-12-15T22:57:45.275060+0000","last_undegraded":"2023-12-15T22:57:45.275060+0000","last_fullsized":"2023-12-15T22:57:45.275060+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4941225,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":26,"num_read_kb":16107,"num_write":17,"num_write_kb":4759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":143,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-12-15T22:57:44.310294+0000","last_change":"2023-12-15T22:57:43.268384+0000","last_active":"2023-12-15T22:57:44.310294+0000","last_peered":"2023-12-15T22:57:44.310294+0000","last_clean":"2023-12-15T22:57:44.310294+0000","last_became_active":"2023-12-15T22:56:39.552856+0000","last_became_peered":"2023-12-15T22:56:39.552856+0000","last_unstale":"2023-12-15T22:57:44.310294+0000","last_undegraded":"2023-12-15T22:57:44.310294+0000","last_fullsized":"2023-12-15T22:57:44.310294+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"78'103","reported_seq":257,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-12-15T22:57:44.366957+0000","last_change":"2023-12-15T22:57:42.259084+0000","last_active":"2023-12-15T22:57:44.366957+0000","last_peered":"2023-12-15T22:57:44.366957+0000","last_clean":"2023-12-15T22:57:44.366957+0000","last_became_active":"2023-12-15T22:56:39.495490+0000","last_became_peered":"2023-12-15T22:56:39.495490+0000","last_unstale":"2023-12-15T22:57:44.366957+0000","last_undegraded":"2023-12-15T22:57:44.366957+0000","last_fullsized":"2023-12-15T22:57:44.366957+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14924583,"num_objects":15,"num_object_clones":7,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":98,"num_read_kb":65688,"num_write":84,"num_write_kb":23617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"71'63","reported_seq":210,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-12-15T22:57:45.270411+0000","last_change":"2023-12-15T22:57:45.281099+0000","last_active":"2023-12-15T22:57:45.270411+0000","last_peered":"2023-12-15T22:57:45.270411+0000","last_clean":"2023-12-15T22:57:45.270411+0000","last_became_active":"2023-12-15T22:56:39.579234+0000","last_became_peered":"2023-12-15T22:56:39.579234+0000","last_unstale":"2023-12-15T22:57:45.270411+0000","last_undegraded":"2023-12-15T22:57:45.270411+0000","last_fullsized":"2023-12-15T22:57:45.270411+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9025326,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":57,"num_read_kb":37632,"num_write":53,"num_write_kb":14095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"64'23","reported_seq":160,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-12-15T22:57:44.325450+0000","last_change":"2023-12-15T22:57:43.541696+0000","last_active":"2023-12-15T22:57:44.325450+0000","last_peered":"2023-12-15T22:57:44.325450+0000","last_clean":"2023-12-15T22:57:44.325450+0000","last_became_active":"2023-12-15T22:56:39.572176+0000","last_became_peered":"2023-12-15T22:56:39.572176+0000","last_unstale":"2023-12-15T22:57:44.325450+0000","last_undegraded":"2023-12-15T22:57:44.325450+0000","last_fullsized":"2023-12-15T22:57:44.325450+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2739541,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":17,"num_read_kb":3686,"num_write":21,"num_write_kb":6135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"79'40","reported_seq":176,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-12-15T22:57:45.273054+0000","last_change":"2023-12-15T22:57:45.299561+0000","last_active":"2023-12-15T22:57:45.273054+0000","last_peered":"2023-12-15T22:57:45.273054+0000","last_clean":"2023-12-15T22:57:45.273054+0000","last_became_active":"2023-12-15T22:56:39.516361+0000","last_became_peered":"2023-12-15T22:56:39.516361+0000","last_unstale":"2023-12-15T22:57:45.273054+0000","last_undegraded":"2023-12-15T22:57:45.273054+0000","last_fullsized":"2023-12-15T22:57:45.273054+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2285719,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":24,"num_read_kb":19607,"num_write":31,"num_write_kb":8847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"79'164","reported_seq":290,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-12-15T22:57:45.275570+0000","last_change":"2023-12-15T22:57:45.301423+0000","last_active":"2023-12-15T22:57:45.275570+0000","last_peered":"2023-12-15T22:57:45.275570+0000","last_clean":"2023-12-15T22:57:45.275570+0000","last_became_active":"2023-12-15T22:56:39.519748+0000","last_became_peered":"2023-12-15T22:56:39.519748+0000","last_unstale":"2023-12-15T22:57:45.275570+0000","last_undegraded":"2023-12-15T22:57:45.275570+0000","last_fullsized":"2023-12-15T22:57:45.275570+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15227109,"num_objects":26,"num_object_clones":16,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":120,"num_read_kb":67758,"num_write":110,"num_write_kb":31583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"69'44","reported_seq":192,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-12-15T22:57:44.291641+0000","last_change":"2023-12-15T22:57:42.257729+0000","last_active":"2023-12-15T22:57:44.291641+0000","last_peered":"2023-12-15T22:57:44.291641+0000","last_clean":"2023-12-15T22:57:44.291641+0000","last_became_active":"2023-12-15T22:56:39.543179+0000","last_became_peered":"2023-12-15T22:56:39.543179+0000","last_unstale":"2023-12-15T22:57:44.291641+0000","last_undegraded":"2023-12-15T22:57:44.291641+0000","last_fullsized":"2023-12-15T22:57:44.291641+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8857223,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":33357,"num_write":40,"num_write_kb":10468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":238,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-12-15T22:57:44.330872+0000","last_change":"2023-12-15T22:56:32.607017+0000","last_active":"2023-12-15T22:57:44.330872+0000","last_peered":"2023-12-15T22:57:44.330872+0000","last_clean":"2023-12-15T22:57:44.330872+0000","last_became_active":"2023-12-15T22:56:32.602384+0000","last_became_peered":"2023-12-15T22:56:32.602384+0000","last_unstale":"2023-12-15T22:57:44.330872+0000","last_undegraded":"2023-12-15T22:57:44.330872+0000","last_fullsized":"2023-12-15T22:57:44.330872+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_clean_scrub_stamp":"2023-12-15T22:56:31.525572+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":115,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-12-15T22:57:44.290521+0000","last_change":"2023-12-15T22:56:33.627481+0000","last_active":"2023-12-15T22:57:44.290521+0000","last_peered":"2023-12-15T22:57:44.290521+0000","last_clean":"2023-12-15T22:57:44.290521+0000","last_became_active":"2023-12-15T22:56:33.624689+0000","last_became_peered":"2023-12-15T22:56:33.624689+0000","last_unstale":"2023-12-15T22:57:44.290521+0000","last_undegraded":"2023-12-15T22:57:44.290521+0000","last_fullsized":"2023-12-15T22:57:44.290521+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"69'64","reported_seq":204,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-12-15T22:57:44.279409+0000","last_change":"2023-12-15T22:57:42.255718+0000","last_active":"2023-12-15T22:57:44.279409+0000","last_peered":"2023-12-15T22:57:44.279409+0000","last_clean":"2023-12-15T22:57:44.279409+0000","last_became_active":"2023-12-15T22:56:39.529359+0000","last_became_peered":"2023-12-15T22:56:39.529359+0000","last_unstale":"2023-12-15T22:57:44.279409+0000","last_undegraded":"2023-12-15T22:57:44.279409+0000","last_fullsized":"2023-12-15T22:57:44.279409+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7176589,"num_objects":8,"num_object_clones":3,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":36,"num_read_kb":10704,"num_write":56,"num_write_kb":16150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":115,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-12-15T22:57:45.276083+0000","last_change":"2023-12-15T22:56:33.655797+0000","last_active":"2023-12-15T22:57:45.276083+0000","last_peered":"2023-12-15T22:57:45.276083+0000","last_clean":"2023-12-15T22:57:45.276083+0000","last_became_active":"2023-12-15T22:56:33.652885+0000","last_became_peered":"2023-12-15T22:56:33.652885+0000","last_unstale":"2023-12-15T22:57:45.276083+0000","last_undegraded":"2023-12-15T22:57:45.276083+0000","last_fullsized":"2023-12-15T22:57:45.276083+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"78'56","reported_seq":197,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-12-15T22:57:44.399110+0000","last_change":"2023-12-15T22:57:43.557769+0000","last_active":"2023-12-15T22:57:44.399110+0000","last_peered":"2023-12-15T22:57:44.399110+0000","last_clean":"2023-12-15T22:57:44.399110+0000","last_became_active":"2023-12-15T22:56:39.543509+0000","last_became_peered":"2023-12-15T22:56:39.543509+0000","last_unstale":"2023-12-15T22:57:44.399110+0000","last_undegraded":"2023-12-15T22:57:44.399110+0000","last_fullsized":"2023-12-15T22:57:44.399110+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12151395,"num_objects":10,"num_object_clones":4,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":53,"num_read_kb":33778,"num_write":43,"num_write_kb":11871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":115,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-12-15T22:57:44.286622+0000","last_change":"2023-12-15T22:56:33.600485+0000","last_active":"2023-12-15T22:57:44.286622+0000","last_peered":"2023-12-15T22:57:44.286622+0000","last_clean":"2023-12-15T22:57:44.286622+0000","last_became_active":"2023-12-15T22:56:33.597320+0000","last_became_peered":"2023-12-15T22:56:33.597320+0000","last_unstale":"2023-12-15T22:57:44.286622+0000","last_undegraded":"2023-12-15T22:57:44.286622+0000","last_fullsized":"2023-12-15T22:57:44.286622+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"55'26","reported_seq":161,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-12-15T22:57:44.323653+0000","last_change":"2023-12-15T22:57:43.543708+0000","last_active":"2023-12-15T22:57:44.323653+0000","last_peered":"2023-12-15T22:57:44.323653+0000","last_clean":"2023-12-15T22:57:44.323653+0000","last_became_active":"2023-12-15T22:56:39.560980+0000","last_became_peered":"2023-12-15T22:56:39.560980+0000","last_unstale":"2023-12-15T22:57:44.323653+0000","last_undegraded":"2023-12-15T22:57:44.323653+0000","last_fullsized":"2023-12-15T22:57:44.323653+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5747961,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":13740,"num_write":21,"num_write_kb":6063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":115,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-12-15T22:57:45.274610+0000","last_change":"2023-12-15T22:56:33.648541+0000","last_active":"2023-12-15T22:57:45.274610+0000","last_peered":"2023-12-15T22:57:45.274610+0000","last_clean":"2023-12-15T22:57:45.274610+0000","last_became_active":"2023-12-15T22:56:33.643681+0000","last_became_peered":"2023-12-15T22:56:33.643681+0000","last_unstale":"2023-12-15T22:57:45.274610+0000","last_undegraded":"2023-12-15T22:57:45.274610+0000","last_fullsized":"2023-12-15T22:57:45.274610+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"61'57","reported_seq":211,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-12-15T22:57:44.306905+0000","last_change":"2023-12-15T22:57:43.268466+0000","last_active":"2023-12-15T22:57:44.306905+0000","last_peered":"2023-12-15T22:57:44.306905+0000","last_clean":"2023-12-15T22:57:44.306905+0000","last_became_active":"2023-12-15T22:56:39.506510+0000","last_became_peered":"2023-12-15T22:56:39.506510+0000","last_unstale":"2023-12-15T22:57:44.306905+0000","last_undegraded":"2023-12-15T22:57:44.306905+0000","last_fullsized":"2023-12-15T22:57:44.306905+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5647119,"num_objects":11,"num_object_clones":6,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":59,"num_read_kb":42594,"num_write":44,"num_write_kb":13017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"50'13","reported_seq":147,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-12-15T22:57:44.275565+0000","last_change":"2023-12-15T22:57:42.259375+0000","last_active":"2023-12-15T22:57:44.275565+0000","last_peered":"2023-12-15T22:57:44.275565+0000","last_clean":"2023-12-15T22:57:44.275565+0000","last_became_active":"2023-12-15T22:56:39.511159+0000","last_became_peered":"2023-12-15T22:56:39.511159+0000","last_unstale":"2023-12-15T22:57:44.275565+0000","last_undegraded":"2023-12-15T22:57:44.275565+0000","last_fullsized":"2023-12-15T22:57:44.275565+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":5,"num_read_kb":2,"num_write":9,"num_write_kb":2133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":119,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-12-15T22:57:44.316826+0000","last_change":"2023-12-15T22:56:33.633890+0000","last_active":"2023-12-15T22:57:44.316826+0000","last_peered":"2023-12-15T22:57:44.316826+0000","last_clean":"2023-12-15T22:57:44.316826+0000","last_became_active":"2023-12-15T22:56:33.631134+0000","last_became_peered":"2023-12-15T22:56:33.631134+0000","last_unstale":"2023-12-15T22:57:44.316826+0000","last_undegraded":"2023-12-15T22:57:44.316826+0000","last_fullsized":"2023-12-15T22:57:44.316826+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"54'47","reported_seq":183,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-12-15T22:57:44.315686+0000","last_change":"2023-12-15T22:57:43.269604+0000","last_active":"2023-12-15T22:57:44.315686+0000","last_peered":"2023-12-15T22:57:44.315686+0000","last_clean":"2023-12-15T22:57:44.315686+0000","last_became_active":"2023-12-15T22:56:39.532730+0000","last_became_peered":"2023-12-15T22:56:39.532730+0000","last_unstale":"2023-12-15T22:57:44.315686+0000","last_undegraded":"2023-12-15T22:57:44.315686+0000","last_fullsized":"2023-12-15T22:57:44.315686+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3764735,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":35,"num_read_kb":28298,"num_write":27,"num_write_kb":8830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":109,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-12-15T22:57:44.315563+0000","last_change":"2023-12-15T22:56:33.655736+0000","last_active":"2023-12-15T22:57:44.315563+0000","last_peered":"2023-12-15T22:57:44.315563+0000","last_clean":"2023-12-15T22:57:44.315563+0000","last_became_active":"2023-12-15T22:56:33.652917+0000","last_became_peered":"2023-12-15T22:56:33.652917+0000","last_unstale":"2023-12-15T22:57:44.315563+0000","last_undegraded":"2023-12-15T22:57:44.315563+0000","last_fullsized":"2023-12-15T22:57:44.315563+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"78'54","reported_seq":201,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-12-15T22:57:44.466182+0000","last_change":"2023-12-15T22:57:43.543265+0000","last_active":"2023-12-15T22:57:44.466182+0000","last_peered":"2023-12-15T22:57:44.466182+0000","last_clean":"2023-12-15T22:57:44.466182+0000","last_became_active":"2023-12-15T22:56:39.557252+0000","last_became_peered":"2023-12-15T22:56:39.557252+0000","last_unstale":"2023-12-15T22:57:44.466182+0000","last_undegraded":"2023-12-15T22:57:44.466182+0000","last_fullsized":"2023-12-15T22:57:44.466182+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5916064,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":53,"num_read_kb":39066,"num_write":46,"num_write_kb":13706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":119,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-12-15T22:57:44.310922+0000","last_change":"2023-12-15T22:56:33.632348+0000","last_active":"2023-12-15T22:57:44.310922+0000","last_peered":"2023-12-15T22:57:44.310922+0000","last_clean":"2023-12-15T22:57:44.310922+0000","last_became_active":"2023-12-15T22:56:33.629022+0000","last_became_peered":"2023-12-15T22:56:33.629022+0000","last_unstale":"2023-12-15T22:57:44.310922+0000","last_undegraded":"2023-12-15T22:57:44.310922+0000","last_fullsized":"2023-12-15T22:57:44.310922+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"70'9","reported_seq":157,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-12-15T22:57:44.309364+0000","last_change":"2023-12-15T22:57:43.270777+0000","last_active":"2023-12-15T22:57:44.309364+0000","last_peered":"2023-12-15T22:57:44.309364+0000","last_clean":"2023-12-15T22:57:44.309364+0000","last_became_active":"2023-12-15T22:56:39.535145+0000","last_became_peered":"2023-12-15T22:56:39.535145+0000","last_unstale":"2023-12-15T22:57:44.309364+0000","last_undegraded":"2023-12-15T22:57:44.309364+0000","last_fullsized":"2023-12-15T22:57:44.309364+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2756348,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":14063,"num_write":8,"num_write_kb":2448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":115,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-12-15T22:57:44.277948+0000","last_change":"2023-12-15T22:56:33.627005+0000","last_active":"2023-12-15T22:57:44.277948+0000","last_peered":"2023-12-15T22:57:44.277948+0000","last_clean":"2023-12-15T22:57:44.277948+0000","last_became_active":"2023-12-15T22:56:33.622181+0000","last_became_peered":"2023-12-15T22:56:33.622181+0000","last_unstale":"2023-12-15T22:57:44.277948+0000","last_undegraded":"2023-12-15T22:57:44.277948+0000","last_fullsized":"2023-12-15T22:57:44.277948+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":101160937,"num_objects":119,"num_object_clones":63,"num_object_copies":357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":665,"num_read_kb":426080,"num_write":610,"num_write_kb":173722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":787,"ondisk_log_size":787,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705739,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":81732,"kb_used_data":54908,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94290108,"statfs":{"total":96636764160,"available":96553070592,"internally_reserved":0,"allocated":56225792,"data_stored":59041714,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70716,"kb_used_data":43892,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94301124,"statfs":{"total":96636764160,"available":96564350976,"internally_reserved":0,"allocated":44945408,"data_stored":46731351,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705738,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75472,"kb_used_data":48648,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94296368,"statfs":{"total":96636764160,"available":96559480832,"internally_reserved":0,"allocated":49815552,"data_stored":51254610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705739,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":89624,"kb_used_data":58640,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94282216,"statfs":{"total":96636764160,"available":96544989184,"internally_reserved":0,"allocated":60047360,"data_stored":61915949,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-15T22:57:47.287 INFO:tasks.rados.rados.0.smithi071.stdout:330: done (1 left) 2023-12-15T22:57:47.287 INFO:tasks.rados.rados.0.smithi071.stdout:331: write (excl) oid 28 current snap is 34 2023-12-15T22:57:47.287 INFO:tasks.rados.rados.0.smithi071.stdout:329: finishing write tid 1 to smithi07183189-29 2023-12-15T22:57:47.287 INFO:tasks.rados.rados.0.smithi071.stdout:331: seq_num 183 ranges {675681=563039,1858793=746713,3025937=49744} 2023-12-15T22:57:47.294 INFO:tasks.rados.rados.0.smithi071.stdout:331: writing smithi07183189-28 from 675681 to 1238720 tid 1 2023-12-15T22:57:47.301 INFO:tasks.rados.rados.0.smithi071.stdout:331: writing smithi07183189-28 from 1858793 to 2605506 tid 2 2023-12-15T22:57:47.304 INFO:tasks.rados.rados.0.smithi071.stdout:331: writing smithi07183189-28 from 3025937 to 3075681 tid 3 2023-12-15T22:57:47.304 INFO:tasks.rados.rados.0.smithi071.stdout:332: write oid 19 current snap is 34 2023-12-15T22:57:47.304 INFO:tasks.rados.rados.0.smithi071.stdout:332: seq_num 184 ranges {692488=638288,1917275=606724} 2023-12-15T22:57:47.311 INFO:tasks.rados.rados.0.smithi071.stdout:332: writing smithi07183189-19 from 692488 to 1330776 tid 1 2023-12-15T22:57:47.323 INFO:tasks.rados.rados.0.smithi071.stdout:332: writing smithi07183189-19 from 1917275 to 2523999 tid 2 2023-12-15T22:57:47.323 INFO:tasks.rados.rados.0.smithi071.stdout:333: read oid 41 snap 32 2023-12-15T22:57:47.323 INFO:tasks.rados.rados.0.smithi071.stdout:333: expect (ObjNum 113 snap 10 seq_num 113) 2023-12-15T22:57:47.323 INFO:tasks.rados.rados.0.smithi071.stdout:334: snap_remove snap 33 2023-12-15T22:57:48.018 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:57:48.123 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:48.122+0000 7f56e402d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:48.126 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:48.126+0000 7f56e402d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:48.293 INFO:tasks.rados.rados.0.smithi071.stdout:334: done (4 left) 2023-12-15T22:57:48.293 INFO:tasks.rados.rados.0.smithi071.stdout:335: write oid 18 current snap is 34 2023-12-15T22:57:48.293 INFO:tasks.rados.rados.0.smithi071.stdout:329: finishing write tid 2 to smithi07183189-29 2023-12-15T22:57:48.293 INFO:tasks.rados.rados.0.smithi071.stdout:335: seq_num 185 ranges {709295=713537,1859404=750382} 2023-12-15T22:57:48.303 INFO:tasks.rados.rados.0.smithi071.stdout:335: writing smithi07183189-18 from 709295 to 1422832 tid 1 2023-12-15T22:57:48.310 INFO:tasks.rados.rados.0.smithi071.stdout:335: writing smithi07183189-18 from 1859404 to 2609786 tid 2 2023-12-15T22:57:48.311 INFO:tasks.rados.rados.0.smithi071.stdout:336: read oid 47 snap -1 2023-12-15T22:57:48.311 INFO:tasks.rados.rados.0.smithi071.stdout:336: expect (ObjNum 163 snap 24 seq_num 163) 2023-12-15T22:57:48.311 INFO:tasks.rados.rados.0.smithi071.stdout:337: write (excl) oid 17 current snap is 34 2023-12-15T22:57:48.311 INFO:tasks.rados.rados.0.smithi071.stdout:329: finishing write tid 3 to smithi07183189-29 2023-12-15T22:57:48.311 INFO:tasks.rados.rados.0.smithi071.stdout:337: seq_num 186 ranges {726102=788786,1917886=610393,3108206=17896} 2023-12-15T22:57:48.321 INFO:tasks.rados.rados.0.smithi071.stdout:337: writing smithi07183189-17 from 726102 to 1514888 tid 1 2023-12-15T22:57:48.328 INFO:tasks.rados.rados.0.smithi071.stdout:337: writing smithi07183189-17 from 1917886 to 2528279 tid 2 2023-12-15T22:57:48.332 INFO:tasks.rados.rados.0.smithi071.stdout:337: writing smithi07183189-17 from 3108206 to 3126102 tid 3 2023-12-15T22:57:48.332 INFO:tasks.rados.rados.0.smithi071.stdout:338: snap_remove snap 28 2023-12-15T22:57:49.303 INFO:tasks.rados.rados.0.smithi071.stdout:338: done (7 left) 2023-12-15T22:57:49.303 INFO:tasks.rados.rados.0.smithi071.stdout:339: read oid 45 snap -1 2023-12-15T22:57:49.303 INFO:tasks.rados.rados.0.smithi071.stdout:339: expect (ObjNum 152 snap 21 seq_num 152) 2023-12-15T22:57:49.303 INFO:tasks.rados.rados.0.smithi071.stdout:340: snap_create 2023-12-15T22:57:49.303 INFO:tasks.rados.rados.0.smithi071.stdout:329: finishing write tid 4 to smithi07183189-29 2023-12-15T22:57:49.303 INFO:tasks.rados.rados.0.smithi071.stdout:329: oid 29 updating version 0 to 166 2023-12-15T22:57:49.304 INFO:tasks.rados.rados.0.smithi071.stdout:329: oid 29 updating version 166 to 167 2023-12-15T22:57:49.304 INFO:tasks.rados.rados.0.smithi071.stdout:329: oid 29 updating version 167 to 168 2023-12-15T22:57:49.304 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 168 (ObjNum 182 snap 34 seq_num 182) dirty exists 2023-12-15T22:57:49.304 INFO:tasks.rados.rados.0.smithi071.stdout:329: left oid 29 (ObjNum 182 snap 34 seq_num 182) 2023-12-15T22:57:49.304 INFO:tasks.rados.rados.0.smithi071.stdout:332: finishing write tid 1 to smithi07183189-19 2023-12-15T22:57:49.304 INFO:tasks.rados.rados.0.smithi071.stdout:333: expect (ObjNum 113 snap 10 seq_num 113) 2023-12-15T22:57:49.513 INFO:tasks.rados.rados.0.smithi071.stdout:331: finishing write tid 1 to smithi07183189-28 2023-12-15T22:57:49.513 INFO:tasks.rados.rados.0.smithi071.stdout:332: finishing write tid 2 to smithi07183189-19 2023-12-15T22:57:49.513 INFO:tasks.rados.rados.0.smithi071.stdout:331: finishing write tid 2 to smithi07183189-28 2023-12-15T22:57:49.513 INFO:tasks.rados.rados.0.smithi071.stdout:332: finishing write tid 3 to smithi07183189-19 2023-12-15T22:57:49.513 INFO:tasks.rados.rados.0.smithi071.stdout:332: finishing write tid 4 to smithi07183189-19 2023-12-15T22:57:49.513 INFO:tasks.rados.rados.0.smithi071.stdout:332: oid 19 updating version 0 to 68 2023-12-15T22:57:49.513 INFO:tasks.rados.rados.0.smithi071.stdout:332: oid 19 version 68 is already newer than 65 2023-12-15T22:57:49.513 INFO:tasks.rados.rados.0.smithi071.stdout:332: oid 19 updating version 68 to 69 2023-12-15T22:57:49.513 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 69 (ObjNum 184 snap 34 seq_num 184) dirty exists 2023-12-15T22:57:49.514 INFO:tasks.rados.rados.0.smithi071.stdout:332: left oid 19 (ObjNum 184 snap 34 seq_num 184) 2023-12-15T22:57:49.514 INFO:tasks.rados.rados.0.smithi071.stdout:331: finishing write tid 3 to smithi07183189-28 2023-12-15T22:57:49.514 INFO:tasks.rados.rados.0.smithi071.stdout:331: finishing write tid 4 to smithi07183189-28 2023-12-15T22:57:49.514 INFO:tasks.rados.rados.0.smithi071.stdout:331: finishing write tid 5 to smithi07183189-28 2023-12-15T22:57:49.514 INFO:tasks.rados.rados.0.smithi071.stdout:331: oid 28 updating version 0 to 33 2023-12-15T22:57:49.514 INFO:tasks.rados.rados.0.smithi071.stdout:331: oid 28 version 33 is already newer than 31 2023-12-15T22:57:49.514 INFO:tasks.rados.rados.0.smithi071.stdout:331: oid 28 version 33 is already newer than 30 2023-12-15T22:57:49.514 INFO:tasks.rados.rados.0.smithi071.stdout:331: oid 28 version 33 is already newer than 33 2023-12-15T22:57:49.514 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 33 (ObjNum 183 snap 34 seq_num 183) dirty exists 2023-12-15T22:57:49.514 INFO:tasks.rados.rados.0.smithi071.stdout:331: left oid 28 (ObjNum 183 snap 34 seq_num 183) 2023-12-15T22:57:49.514 INFO:tasks.rados.rados.0.smithi071.stdout:336: expect (ObjNum 163 snap 24 seq_num 163) 2023-12-15T22:57:49.621 INFO:tasks.rados.rados.0.smithi071.stdout:337: finishing write tid 1 to smithi07183189-17 2023-12-15T22:57:49.621 INFO:tasks.rados.rados.0.smithi071.stdout:335: finishing write tid 1 to smithi07183189-18 2023-12-15T22:57:49.621 INFO:tasks.rados.rados.0.smithi071.stdout:335: finishing write tid 2 to smithi07183189-18 2023-12-15T22:57:49.621 INFO:tasks.rados.rados.0.smithi071.stdout:337: finishing write tid 2 to smithi07183189-17 2023-12-15T22:57:49.622 INFO:tasks.rados.rados.0.smithi071.stdout:335: finishing write tid 3 to smithi07183189-18 2023-12-15T22:57:49.622 INFO:tasks.rados.rados.0.smithi071.stdout:335: finishing write tid 4 to smithi07183189-18 2023-12-15T22:57:49.622 INFO:tasks.rados.rados.0.smithi071.stdout:335: oid 18 updating version 0 to 45 2023-12-15T22:57:49.622 INFO:tasks.rados.rados.0.smithi071.stdout:335: oid 18 updating version 45 to 46 2023-12-15T22:57:49.622 INFO:tasks.rados.rados.0.smithi071.stdout:335: oid 18 version 46 is already newer than 44 2023-12-15T22:57:49.622 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 46 (ObjNum 185 snap 34 seq_num 185) dirty exists 2023-12-15T22:57:49.622 INFO:tasks.rados.rados.0.smithi071.stdout:335: left oid 18 (ObjNum 185 snap 34 seq_num 185) 2023-12-15T22:57:49.622 INFO:tasks.rados.rados.0.smithi071.stdout:337: finishing write tid 3 to smithi07183189-17 2023-12-15T22:57:49.622 INFO:tasks.rados.rados.0.smithi071.stdout:337: finishing write tid 4 to smithi07183189-17 2023-12-15T22:57:49.622 INFO:tasks.rados.rados.0.smithi071.stdout:337: finishing write tid 5 to smithi07183189-17 2023-12-15T22:57:49.622 INFO:tasks.rados.rados.0.smithi071.stdout:337: oid 17 updating version 0 to 110 2023-12-15T22:57:49.622 INFO:tasks.rados.rados.0.smithi071.stdout:337: oid 17 version 110 is already newer than 107 2023-12-15T22:57:49.622 INFO:tasks.rados.rados.0.smithi071.stdout:337: oid 17 version 110 is already newer than 108 2023-12-15T22:57:49.622 INFO:tasks.rados.rados.0.smithi071.stdout:337: oid 17 version 110 is already newer than 109 2023-12-15T22:57:49.622 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 17 v 110 (ObjNum 186 snap 34 seq_num 186) dirty exists 2023-12-15T22:57:49.622 INFO:tasks.rados.rados.0.smithi071.stdout:337: left oid 17 (ObjNum 186 snap 34 seq_num 186) 2023-12-15T22:57:49.623 INFO:tasks.rados.rados.0.smithi071.stdout:339: expect (ObjNum 152 snap 21 seq_num 152) 2023-12-15T22:57:50.195 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:57:50.300 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:50.299+0000 7fe92271e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:50.304 INFO:tasks.rados.rados.0.smithi071.stdout:329: done (8 left) 2023-12-15T22:57:50.304 INFO:tasks.rados.rados.0.smithi071.stdout:331: done (7 left) 2023-12-15T22:57:50.304 INFO:tasks.rados.rados.0.smithi071.stdout:332: done (6 left) 2023-12-15T22:57:50.304 INFO:tasks.rados.rados.0.smithi071.stdout:333: done (5 left) 2023-12-15T22:57:50.304 INFO:tasks.rados.rados.0.smithi071.stdout:335: done (4 left) 2023-12-15T22:57:50.304 INFO:tasks.rados.rados.0.smithi071.stdout:336: done (3 left) 2023-12-15T22:57:50.304 INFO:tasks.rados.rados.0.smithi071.stdout:337: done (2 left) 2023-12-15T22:57:50.304 INFO:tasks.rados.rados.0.smithi071.stdout:339: done (1 left) 2023-12-15T22:57:50.304 INFO:tasks.rados.rados.0.smithi071.stdout:340: done (0 left) 2023-12-15T22:57:50.304 INFO:tasks.rados.rados.0.smithi071.stdout:341: snap_create 2023-12-15T22:57:50.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:50.304+0000 7fe92271e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:51.305 INFO:tasks.rados.rados.0.smithi071.stdout:341: done (0 left) 2023-12-15T22:57:51.305 INFO:tasks.rados.rados.0.smithi071.stdout:342: read oid 24 snap 32 2023-12-15T22:57:51.305 INFO:tasks.rados.rados.0.smithi071.stdout:342: expect deleted 2023-12-15T22:57:51.305 INFO:tasks.rados.rados.0.smithi071.stdout:343: delete oid 27 current snap is 36 2023-12-15T22:57:51.363 INFO:tasks.rados.rados.0.smithi071.stdout:343: done (1 left) 2023-12-15T22:57:51.363 INFO:tasks.rados.rados.0.smithi071.stdout:344: write (excl) oid 35 current snap is 36 2023-12-15T22:57:51.363 INFO:tasks.rados.rados.0.smithi071.stdout:344: seq_num 188 ranges {759716=539284,2034850=614062} 2023-12-15T22:57:51.370 INFO:tasks.rados.rados.0.smithi071.stdout:344: writing smithi07183189-35 from 759716 to 1299000 tid 1 2023-12-15T22:57:51.372 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T22:57:51.373 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T22:57:51.373 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T22:57:51.378 INFO:tasks.rados.rados.0.smithi071.stdout:344: writing smithi07183189-35 from 2034850 to 2648912 tid 2 2023-12-15T22:57:51.378 INFO:tasks.rados.rados.0.smithi071.stdout:345: snap_create 2023-12-15T22:57:51.428 INFO:tasks.rados.rados.0.smithi071.stdout:344: finishing write tid 1 to smithi07183189-35 2023-12-15T22:57:51.448 INFO:tasks.rados.rados.0.smithi071.stdout:344: finishing write tid 2 to smithi07183189-35 2023-12-15T22:57:51.460 INFO:tasks.rados.rados.0.smithi071.stdout:344: finishing write tid 3 to smithi07183189-35 2023-12-15T22:57:51.462 INFO:tasks.rados.rados.0.smithi071.stdout:344: finishing write tid 4 to smithi07183189-35 2023-12-15T22:57:51.462 INFO:tasks.rados.rados.0.smithi071.stdout:344: oid 35 updating version 0 to 75 2023-12-15T22:57:51.462 INFO:tasks.rados.rados.0.smithi071.stdout:344: oid 35 updating version 75 to 76 2023-12-15T22:57:51.462 INFO:tasks.rados.rados.0.smithi071.stdout:344: oid 35 updating version 76 to 77 2023-12-15T22:57:51.462 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 77 (ObjNum 188 snap 36 seq_num 188) dirty exists 2023-12-15T22:57:51.462 INFO:tasks.rados.rados.0.smithi071.stdout:344: left oid 35 (ObjNum 188 snap 36 seq_num 188) 2023-12-15T22:57:51.486 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:51.483+0000 7fb83b79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:51.489 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:51.489+0000 7fb83b79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:51.679 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:57:51.679 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-12-15T22:57:51.691 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":119,"stamp":"2023-12-15T22:57:51.303413+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":106524559,"num_objects":125,"num_object_clones":67,"num_object_copies":375,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":19,"num_read":794,"num_read_kb":447618,"num_write":769,"num_write_kb":182321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":947,"ondisk_log_size":947,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":324656,"kb_used_data":213200,"kb_used_omap":0,"kb_used_meta":111424,"kb_avail":377162704,"statfs":{"total":386547056640,"available":386214608896,"internally_reserved":0,"allocated":218316800,"data_stored":223509692,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":114098176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4773254,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":26,"num_read_kb":21447,"num_write":22,"num_write_kb":6705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.017750"},"pg_stats":[{"pgid":"3.f","version":"73'24","reported_seq":176,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-12-15T22:57:49.321571+0000","last_change":"2023-12-15T22:57:49.325114+0000","last_active":"2023-12-15T22:57:49.321571+0000","last_peered":"2023-12-15T22:57:49.321571+0000","last_clean":"2023-12-15T22:57:49.321571+0000","last_became_active":"2023-12-15T22:56:39.503797+0000","last_became_peered":"2023-12-15T22:56:39.503797+0000","last_unstale":"2023-12-15T22:57:49.321571+0000","last_undegraded":"2023-12-15T22:57:49.321571+0000","last_fullsized":"2023-12-15T22:57:49.321571+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4941225,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":26,"num_read_kb":16107,"num_write":17,"num_write_kb":4759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":155,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-12-15T22:57:49.329970+0000","last_change":"2023-12-15T22:57:49.333141+0000","last_active":"2023-12-15T22:57:49.329970+0000","last_peered":"2023-12-15T22:57:49.329970+0000","last_clean":"2023-12-15T22:57:49.329970+0000","last_became_active":"2023-12-15T22:56:39.552856+0000","last_became_peered":"2023-12-15T22:56:39.552856+0000","last_unstale":"2023-12-15T22:57:49.329970+0000","last_undegraded":"2023-12-15T22:57:49.329970+0000","last_fullsized":"2023-12-15T22:57:49.329970+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"82'110","reported_seq":269,"reported_epoch":82,"state":"active+clean+snaptrim","last_fresh":"2023-12-15T22:57:48.450905+0000","last_change":"2023-12-15T22:57:48.317183+0000","last_active":"2023-12-15T22:57:48.450905+0000","last_peered":"2023-12-15T22:57:48.450905+0000","last_clean":"2023-12-15T22:57:48.450905+0000","last_became_active":"2023-12-15T22:56:39.495490+0000","last_became_peered":"2023-12-15T22:56:39.495490+0000","last_unstale":"2023-12-15T22:57:48.450905+0000","last_undegraded":"2023-12-15T22:57:48.450905+0000","last_fullsized":"2023-12-15T22:57:48.450905+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15159881,"num_objects":16,"num_object_clones":8,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":100,"num_read_kb":65689,"num_write":89,"num_write_kb":25003,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"83'73","reported_seq":225,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-12-15T22:57:49.314435+0000","last_change":"2023-12-15T22:57:49.359987+0000","last_active":"2023-12-15T22:57:49.314435+0000","last_peered":"2023-12-15T22:57:49.314435+0000","last_clean":"2023-12-15T22:57:49.314435+0000","last_became_active":"2023-12-15T22:56:39.579234+0000","last_became_peered":"2023-12-15T22:56:39.579234+0000","last_unstale":"2023-12-15T22:57:49.314435+0000","last_undegraded":"2023-12-15T22:57:49.314435+0000","last_fullsized":"2023-12-15T22:57:49.314435+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9798448,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":58,"num_read_kb":37633,"num_write":57,"num_write_kb":15312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"84'35","reported_seq":182,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-15T22:57:50.322595+0000","last_change":"2023-12-15T22:57:50.345649+0000","last_active":"2023-12-15T22:57:50.322595+0000","last_peered":"2023-12-15T22:57:50.322595+0000","last_clean":"2023-12-15T22:57:50.322595+0000","last_became_active":"2023-12-15T22:56:39.572176+0000","last_became_peered":"2023-12-15T22:56:39.572176+0000","last_unstale":"2023-12-15T22:57:50.322595+0000","last_undegraded":"2023-12-15T22:57:50.322595+0000","last_fullsized":"2023-12-15T22:57:50.322595+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5815222,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":25,"num_read_kb":8581,"num_write":26,"num_write_kb":7464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"83'50","reported_seq":191,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-12-15T22:57:49.317204+0000","last_change":"2023-12-15T22:57:49.364595+0000","last_active":"2023-12-15T22:57:49.317204+0000","last_peered":"2023-12-15T22:57:49.317204+0000","last_clean":"2023-12-15T22:57:49.317204+0000","last_became_active":"2023-12-15T22:56:39.516361+0000","last_became_peered":"2023-12-15T22:56:39.516361+0000","last_unstale":"2023-12-15T22:57:49.317204+0000","last_undegraded":"2023-12-15T22:57:49.317204+0000","last_fullsized":"2023-12-15T22:57:49.317204+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3109295,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":25,"num_read_kb":19608,"num_write":35,"num_write_kb":10277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"83'180","reported_seq":308,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-12-15T22:57:49.351978+0000","last_change":"2023-12-15T22:57:49.363321+0000","last_active":"2023-12-15T22:57:49.351978+0000","last_peered":"2023-12-15T22:57:49.351978+0000","last_clean":"2023-12-15T22:57:49.351978+0000","last_became_active":"2023-12-15T22:56:39.519748+0000","last_became_peered":"2023-12-15T22:56:39.519748+0000","last_unstale":"2023-12-15T22:57:49.351978+0000","last_undegraded":"2023-12-15T22:57:49.351978+0000","last_fullsized":"2023-12-15T22:57:49.351978+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15092686,"num_objects":27,"num_object_clones":17,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":128,"num_read_kb":74395,"num_write":114,"num_write_kb":32926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"69'44","reported_seq":199,"reported_epoch":82,"state":"active+clean+snaptrim","last_fresh":"2023-12-15T22:57:48.326054+0000","last_change":"2023-12-15T22:57:48.326054+0000","last_active":"2023-12-15T22:57:48.326054+0000","last_peered":"2023-12-15T22:57:48.326054+0000","last_clean":"2023-12-15T22:57:48.326054+0000","last_became_active":"2023-12-15T22:56:39.543179+0000","last_became_peered":"2023-12-15T22:56:39.543179+0000","last_unstale":"2023-12-15T22:57:48.326054+0000","last_undegraded":"2023-12-15T22:57:48.326054+0000","last_fullsized":"2023-12-15T22:57:48.326054+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8857223,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":33357,"num_write":40,"num_write_kb":10468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":248,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-15T22:57:50.322283+0000","last_change":"2023-12-15T22:56:32.607017+0000","last_active":"2023-12-15T22:57:50.322283+0000","last_peered":"2023-12-15T22:57:50.322283+0000","last_clean":"2023-12-15T22:57:50.322283+0000","last_became_active":"2023-12-15T22:56:32.602384+0000","last_became_peered":"2023-12-15T22:56:32.602384+0000","last_unstale":"2023-12-15T22:57:50.322283+0000","last_undegraded":"2023-12-15T22:57:50.322283+0000","last_fullsized":"2023-12-15T22:57:50.322283+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_clean_scrub_stamp":"2023-12-15T22:56:31.525572+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":121,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-12-15T22:57:48.326873+0000","last_change":"2023-12-15T22:56:33.627481+0000","last_active":"2023-12-15T22:57:48.326873+0000","last_peered":"2023-12-15T22:57:48.326873+0000","last_clean":"2023-12-15T22:57:48.326873+0000","last_became_active":"2023-12-15T22:56:33.624689+0000","last_became_peered":"2023-12-15T22:56:33.624689+0000","last_unstale":"2023-12-15T22:57:48.326873+0000","last_undegraded":"2023-12-15T22:57:48.326873+0000","last_fullsized":"2023-12-15T22:57:48.326873+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"69'64","reported_seq":213,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-12-15T22:57:48.349606+0000","last_change":"2023-12-15T22:57:48.352005+0000","last_active":"2023-12-15T22:57:48.349606+0000","last_peered":"2023-12-15T22:57:48.349606+0000","last_clean":"2023-12-15T22:57:48.349606+0000","last_became_active":"2023-12-15T22:56:39.529359+0000","last_became_peered":"2023-12-15T22:56:39.529359+0000","last_unstale":"2023-12-15T22:57:48.349606+0000","last_undegraded":"2023-12-15T22:57:48.349606+0000","last_fullsized":"2023-12-15T22:57:48.349606+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7176589,"num_objects":8,"num_object_clones":3,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":36,"num_read_kb":10704,"num_write":56,"num_write_kb":16150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":123,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-12-15T22:57:49.313424+0000","last_change":"2023-12-15T22:56:33.655797+0000","last_active":"2023-12-15T22:57:49.313424+0000","last_peered":"2023-12-15T22:57:49.313424+0000","last_clean":"2023-12-15T22:57:49.313424+0000","last_became_active":"2023-12-15T22:56:33.652885+0000","last_became_peered":"2023-12-15T22:56:33.652885+0000","last_unstale":"2023-12-15T22:57:49.313424+0000","last_undegraded":"2023-12-15T22:57:49.313424+0000","last_fullsized":"2023-12-15T22:57:49.313424+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"78'56","reported_seq":214,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-15T22:57:50.326227+0000","last_change":"2023-12-15T22:57:50.331657+0000","last_active":"2023-12-15T22:57:50.326227+0000","last_peered":"2023-12-15T22:57:50.326227+0000","last_clean":"2023-12-15T22:57:50.326227+0000","last_became_active":"2023-12-15T22:56:39.543509+0000","last_became_peered":"2023-12-15T22:56:39.543509+0000","last_unstale":"2023-12-15T22:57:50.326227+0000","last_undegraded":"2023-12-15T22:57:50.326227+0000","last_fullsized":"2023-12-15T22:57:50.326227+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12151395,"num_objects":10,"num_object_clones":4,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":59,"num_read_kb":43690,"num_write":43,"num_write_kb":11871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":121,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-12-15T22:57:48.302304+0000","last_change":"2023-12-15T22:56:33.600485+0000","last_active":"2023-12-15T22:57:48.302304+0000","last_peered":"2023-12-15T22:57:48.302304+0000","last_clean":"2023-12-15T22:57:48.302304+0000","last_became_active":"2023-12-15T22:56:33.597320+0000","last_became_peered":"2023-12-15T22:56:33.597320+0000","last_unstale":"2023-12-15T22:57:48.302304+0000","last_undegraded":"2023-12-15T22:57:48.302304+0000","last_fullsized":"2023-12-15T22:57:48.302304+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"55'26","reported_seq":175,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-15T22:57:50.320952+0000","last_change":"2023-12-15T22:57:50.328758+0000","last_active":"2023-12-15T22:57:50.320952+0000","last_peered":"2023-12-15T22:57:50.320952+0000","last_clean":"2023-12-15T22:57:50.320952+0000","last_became_active":"2023-12-15T22:56:39.560980+0000","last_became_peered":"2023-12-15T22:56:39.560980+0000","last_unstale":"2023-12-15T22:57:50.320952+0000","last_undegraded":"2023-12-15T22:57:50.320952+0000","last_fullsized":"2023-12-15T22:57:50.320952+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5747961,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":13740,"num_write":21,"num_write_kb":6063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":123,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-12-15T22:57:49.313745+0000","last_change":"2023-12-15T22:56:33.648541+0000","last_active":"2023-12-15T22:57:49.313745+0000","last_peered":"2023-12-15T22:57:49.313745+0000","last_clean":"2023-12-15T22:57:49.313745+0000","last_became_active":"2023-12-15T22:56:33.643681+0000","last_became_peered":"2023-12-15T22:56:33.643681+0000","last_unstale":"2023-12-15T22:57:49.313745+0000","last_undegraded":"2023-12-15T22:57:49.313745+0000","last_fullsized":"2023-12-15T22:57:49.313745+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"81'63","reported_seq":221,"reported_epoch":83,"state":"active+clean+snaptrim","last_fresh":"2023-12-15T22:57:49.319275+0000","last_change":"2023-12-15T22:57:49.319275+0000","last_active":"2023-12-15T22:57:49.319275+0000","last_peered":"2023-12-15T22:57:49.319275+0000","last_clean":"2023-12-15T22:57:49.319275+0000","last_became_active":"2023-12-15T22:56:39.506510+0000","last_became_peered":"2023-12-15T22:56:39.506510+0000","last_unstale":"2023-12-15T22:57:49.319275+0000","last_undegraded":"2023-12-15T22:57:49.319275+0000","last_fullsized":"2023-12-15T22:57:49.319275+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5647119,"num_objects":11,"num_object_clones":6,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":59,"num_read_kb":42594,"num_write":44,"num_write_kb":13017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"50'13","reported_seq":156,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-12-15T22:57:48.336409+0000","last_change":"2023-12-15T22:57:48.363164+0000","last_active":"2023-12-15T22:57:48.336409+0000","last_peered":"2023-12-15T22:57:48.336409+0000","last_clean":"2023-12-15T22:57:48.336409+0000","last_became_active":"2023-12-15T22:56:39.511159+0000","last_became_peered":"2023-12-15T22:56:39.511159+0000","last_unstale":"2023-12-15T22:57:48.336409+0000","last_undegraded":"2023-12-15T22:57:48.336409+0000","last_fullsized":"2023-12-15T22:57:48.336409+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":5,"num_read_kb":2,"num_write":9,"num_write_kb":2133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":127,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-12-15T22:57:49.316957+0000","last_change":"2023-12-15T22:56:33.633890+0000","last_active":"2023-12-15T22:57:49.316957+0000","last_peered":"2023-12-15T22:57:49.316957+0000","last_clean":"2023-12-15T22:57:49.316957+0000","last_became_active":"2023-12-15T22:56:33.631134+0000","last_became_peered":"2023-12-15T22:56:33.631134+0000","last_unstale":"2023-12-15T22:57:49.316957+0000","last_undegraded":"2023-12-15T22:57:49.316957+0000","last_fullsized":"2023-12-15T22:57:49.316957+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"54'47","reported_seq":195,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-12-15T22:57:49.330599+0000","last_change":"2023-12-15T22:57:49.334146+0000","last_active":"2023-12-15T22:57:49.330599+0000","last_peered":"2023-12-15T22:57:49.330599+0000","last_clean":"2023-12-15T22:57:49.330599+0000","last_became_active":"2023-12-15T22:56:39.532730+0000","last_became_peered":"2023-12-15T22:56:39.532730+0000","last_unstale":"2023-12-15T22:57:49.330599+0000","last_undegraded":"2023-12-15T22:57:49.330599+0000","last_fullsized":"2023-12-15T22:57:49.330599+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3764735,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":35,"num_read_kb":28298,"num_write":27,"num_write_kb":8830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":119,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-15T22:57:50.321702+0000","last_change":"2023-12-15T22:56:33.655736+0000","last_active":"2023-12-15T22:57:50.321702+0000","last_peered":"2023-12-15T22:57:50.321702+0000","last_clean":"2023-12-15T22:57:50.321702+0000","last_became_active":"2023-12-15T22:56:33.652917+0000","last_became_peered":"2023-12-15T22:56:33.652917+0000","last_unstale":"2023-12-15T22:57:50.321702+0000","last_undegraded":"2023-12-15T22:57:50.321702+0000","last_fullsized":"2023-12-15T22:57:50.321702+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"81'56","reported_seq":215,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-15T22:57:50.324829+0000","last_change":"2023-12-15T22:57:50.328375+0000","last_active":"2023-12-15T22:57:50.324829+0000","last_peered":"2023-12-15T22:57:50.324829+0000","last_clean":"2023-12-15T22:57:50.324829+0000","last_became_active":"2023-12-15T22:56:39.557252+0000","last_became_peered":"2023-12-15T22:56:39.557252+0000","last_unstale":"2023-12-15T22:57:50.324829+0000","last_undegraded":"2023-12-15T22:57:50.324829+0000","last_fullsized":"2023-12-15T22:57:50.324829+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5916064,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":53,"num_read_kb":39066,"num_write":46,"num_write_kb":13706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":127,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-12-15T22:57:49.322167+0000","last_change":"2023-12-15T22:56:33.632348+0000","last_active":"2023-12-15T22:57:49.322167+0000","last_peered":"2023-12-15T22:57:49.322167+0000","last_clean":"2023-12-15T22:57:49.322167+0000","last_became_active":"2023-12-15T22:56:33.629022+0000","last_became_peered":"2023-12-15T22:56:33.629022+0000","last_unstale":"2023-12-15T22:57:49.322167+0000","last_undegraded":"2023-12-15T22:57:49.322167+0000","last_fullsized":"2023-12-15T22:57:49.322167+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"70'9","reported_seq":169,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-12-15T22:57:49.328445+0000","last_change":"2023-12-15T22:57:49.332461+0000","last_active":"2023-12-15T22:57:49.328445+0000","last_peered":"2023-12-15T22:57:49.328445+0000","last_clean":"2023-12-15T22:57:49.328445+0000","last_became_active":"2023-12-15T22:56:39.535145+0000","last_became_peered":"2023-12-15T22:56:39.535145+0000","last_unstale":"2023-12-15T22:57:49.328445+0000","last_undegraded":"2023-12-15T22:57:49.328445+0000","last_fullsized":"2023-12-15T22:57:49.328445+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2756348,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":14063,"num_write":8,"num_write_kb":2448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":121,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-12-15T22:57:48.330653+0000","last_change":"2023-12-15T22:56:33.627005+0000","last_active":"2023-12-15T22:57:48.330653+0000","last_peered":"2023-12-15T22:57:48.330653+0000","last_clean":"2023-12-15T22:57:48.330653+0000","last_became_active":"2023-12-15T22:56:33.622181+0000","last_became_peered":"2023-12-15T22:56:33.622181+0000","last_unstale":"2023-12-15T22:57:48.330653+0000","last_undegraded":"2023-12-15T22:57:48.330653+0000","last_fullsized":"2023-12-15T22:57:48.330653+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":105934191,"num_objects":123,"num_object_clones":67,"num_object_copies":369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":19,"num_read":691,"num_read_kb":447527,"num_write":632,"num_write_kb":180427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":850,"ondisk_log_size":850,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705744,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":81892,"kb_used_data":55068,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94289948,"statfs":{"total":96636764160,"available":96552906752,"internally_reserved":0,"allocated":56389632,"data_stored":57095756,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705743,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":72340,"kb_used_data":45516,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94299500,"statfs":{"total":96636764160,"available":96562688000,"internally_reserved":0,"allocated":46608384,"data_stored":47908086,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705743,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":78824,"kb_used_data":52000,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94293016,"statfs":{"total":96636764160,"available":96556048384,"internally_reserved":0,"allocated":53248000,"data_stored":54565417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705744,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":91600,"kb_used_data":60616,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94280240,"statfs":{"total":96636764160,"available":96542965760,"internally_reserved":0,"allocated":62070784,"data_stored":63940433,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-15T22:57:52.239 INFO:tasks.rados.rados.0.smithi071.stdout:342: done (2 left) 2023-12-15T22:57:52.239 INFO:tasks.rados.rados.0.smithi071.stdout:344: done (1 left) 2023-12-15T22:57:52.239 INFO:tasks.rados.rados.0.smithi071.stdout:345: done (0 left) 2023-12-15T22:57:52.239 INFO:tasks.rados.rados.0.smithi071.stdout:346: write (excl) oid 13 current snap is 37 2023-12-15T22:57:52.239 INFO:tasks.rados.rados.0.smithi071.stdout:346: seq_num 189 ranges {776523=614533,1976979=757720} 2023-12-15T22:57:52.248 INFO:tasks.rados.rados.0.smithi071.stdout:346: writing smithi07183189-13 from 776523 to 1391056 tid 1 2023-12-15T22:57:52.258 INFO:tasks.rados.rados.0.smithi071.stdout:346: writing smithi07183189-13 from 1976979 to 2734699 tid 2 2023-12-15T22:57:52.258 INFO:tasks.rados.rados.0.smithi071.stdout:347: snap_create 2023-12-15T22:57:52.298 INFO:tasks.rados.rados.0.smithi071.stdout:346: finishing write tid 1 to smithi07183189-13 2023-12-15T22:57:52.320 INFO:tasks.rados.rados.0.smithi071.stdout:346: finishing write tid 2 to smithi07183189-13 2023-12-15T22:57:52.330 INFO:tasks.rados.rados.0.smithi071.stdout:346: finishing write tid 3 to smithi07183189-13 2023-12-15T22:57:52.331 INFO:tasks.rados.rados.0.smithi071.stdout:346: finishing write tid 4 to smithi07183189-13 2023-12-15T22:57:52.331 INFO:tasks.rados.rados.0.smithi071.stdout:346: oid 13 updating version 0 to 59 2023-12-15T22:57:52.331 INFO:tasks.rados.rados.0.smithi071.stdout:346: oid 13 updating version 59 to 60 2023-12-15T22:57:52.331 INFO:tasks.rados.rados.0.smithi071.stdout:346: oid 13 version 60 is already newer than 58 2023-12-15T22:57:52.331 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 60 (ObjNum 189 snap 37 seq_num 189) dirty exists 2023-12-15T22:57:52.331 INFO:tasks.rados.rados.0.smithi071.stdout:346: left oid 13 (ObjNum 189 snap 37 seq_num 189) 2023-12-15T22:57:52.376 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:57:52.479 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:52.477+0000 7fa0724ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:52.481 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:52.482+0000 7fa0724ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:53.245 INFO:tasks.rados.rados.0.smithi071.stdout:346: done (1 left) 2023-12-15T22:57:53.246 INFO:tasks.rados.rados.0.smithi071.stdout:347: done (0 left) 2023-12-15T22:57:53.246 INFO:tasks.rados.rados.0.smithi071.stdout:348: read oid 49 snap -1 2023-12-15T22:57:53.246 INFO:tasks.rados.rados.0.smithi071.stdout:348: expect (ObjNum 143 snap 18 seq_num 143) 2023-12-15T22:57:53.246 INFO:tasks.rados.rados.0.smithi071.stdout:349: read oid 33 snap 37 2023-12-15T22:57:53.246 INFO:tasks.rados.rados.0.smithi071.stdout:349: expect deleted 2023-12-15T22:57:53.246 INFO:tasks.rados.rados.0.smithi071.stdout:350: read oid 38 snap -1 2023-12-15T22:57:53.246 INFO:tasks.rados.rados.0.smithi071.stdout:350: expect (ObjNum 54 snap 0 seq_num 54) 2023-12-15T22:57:53.246 INFO:tasks.rados.rados.0.smithi071.stdout:351: snap_remove snap 35 2023-12-15T22:57:54.247 INFO:tasks.rados.rados.0.smithi071.stdout:351: done (3 left) 2023-12-15T22:57:54.247 INFO:tasks.rados.rados.0.smithi071.stdout:352: snap_remove snap 31 2023-12-15T22:57:54.559 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:57:54.675 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:54.674+0000 7f903e79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:54.678 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:54.678+0000 7f903e79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:55.253 INFO:tasks.rados.rados.0.smithi071.stdout:352: done (3 left) 2023-12-15T22:57:55.253 INFO:tasks.rados.rados.0.smithi071.stdout:353: snap_remove snap 36 2023-12-15T22:57:56.256 INFO:tasks.rados.rados.0.smithi071.stdout:353: done (3 left) 2023-12-15T22:57:56.256 INFO:tasks.rados.rados.0.smithi071.stdout:354: delete oid 15 current snap is 38 2023-12-15T22:57:56.256 INFO:tasks.rados.rados.0.smithi071.stdout:348: expect (ObjNum 143 snap 18 seq_num 143) 2023-12-15T22:57:56.391 INFO:tasks.rados.rados.0.smithi071.stdout:350: expect (ObjNum 54 snap 0 seq_num 54) 2023-12-15T22:57:56.542 INFO:tasks.rados.rados.0.smithi071.stdout:348: done (3 left) 2023-12-15T22:57:56.542 INFO:tasks.rados.rados.0.smithi071.stdout:349: done (2 left) 2023-12-15T22:57:56.542 INFO:tasks.rados.rados.0.smithi071.stdout:350: done (1 left) 2023-12-15T22:57:56.542 INFO:tasks.rados.rados.0.smithi071.stdout:354: done (0 left) 2023-12-15T22:57:56.542 INFO:tasks.rados.rados.0.smithi071.stdout:355: write (excl) oid 24 current snap is 38 2023-12-15T22:57:56.542 INFO:tasks.rados.rados.0.smithi071.stdout:355: seq_num 191 ranges {410137=481384,1410296=761389,2728968=481169} 2023-12-15T22:57:56.545 INFO:tasks.rados.rados.0.smithi071.stdout:355: writing smithi07183189-24 from 410137 to 891521 tid 1 2023-12-15T22:57:56.554 INFO:tasks.rados.rados.0.smithi071.stdout:355: writing smithi07183189-24 from 1410296 to 2171685 tid 2 2023-12-15T22:57:56.561 INFO:tasks.rados.rados.0.smithi071.stdout:355: writing smithi07183189-24 from 2728968 to 3210137 tid 3 2023-12-15T22:57:56.561 INFO:tasks.rados.rados.0.smithi071.stdout:356: read oid 7 snap -1 2023-12-15T22:57:56.561 INFO:tasks.rados.rados.0.smithi071.stdout:356: expect (ObjNum 167 snap 26 seq_num 167) 2023-12-15T22:57:56.562 INFO:tasks.rados.rados.0.smithi071.stdout:357: read oid 14 snap -1 2023-12-15T22:57:56.562 INFO:tasks.rados.rados.0.smithi071.stdout:357: expect (ObjNum 140 snap 17 seq_num 140) 2023-12-15T22:57:56.562 INFO:tasks.rados.rados.0.smithi071.stdout:358: write oid 36 current snap is 38 2023-12-15T22:57:56.562 INFO:tasks.rados.rados.0.smithi071.stdout:358: seq_num 192 ranges {426944=556633,1468778=621400,2872744=354200} 2023-12-15T22:57:56.564 INFO:tasks.rados.rados.0.smithi071.stdout:358: writing smithi07183189-36 from 426944 to 983577 tid 1 2023-12-15T22:57:56.570 INFO:tasks.rados.rados.0.smithi071.stdout:358: writing smithi07183189-36 from 1468778 to 2090178 tid 2 2023-12-15T22:57:56.578 INFO:tasks.rados.rados.0.smithi071.stdout:358: writing smithi07183189-36 from 2872744 to 3226944 tid 3 2023-12-15T22:57:56.579 INFO:tasks.rados.rados.0.smithi071.stdout:359: read oid 4 snap -1 2023-12-15T22:57:56.579 INFO:tasks.rados.rados.0.smithi071.stdout:359: expect (ObjNum 117 snap 10 seq_num 117) 2023-12-15T22:57:56.579 INFO:tasks.rados.rados.0.smithi071.stdout:360: read oid 45 snap -1 2023-12-15T22:57:56.579 INFO:tasks.rados.rados.0.smithi071.stdout:360: expect (ObjNum 152 snap 21 seq_num 152) 2023-12-15T22:57:56.579 INFO:tasks.rados.rados.0.smithi071.stdout:361: write oid 47 current snap is 38 2023-12-15T22:57:56.579 INFO:tasks.rados.rados.0.smithi071.stdout:355: finishing write tid 1 to smithi07183189-24 2023-12-15T22:57:56.579 INFO:tasks.rados.rados.0.smithi071.stdout:361: seq_num 193 ranges {443751=631882,1810907=765058,3067461=176290} 2023-12-15T22:57:56.584 INFO:tasks.rados.rados.0.smithi071.stdout:361: writing smithi07183189-47 from 443751 to 1075633 tid 1 2023-12-15T22:57:56.592 INFO:tasks.rados.rados.0.smithi071.stdout:361: writing smithi07183189-47 from 1810907 to 2575965 tid 2 2023-12-15T22:57:56.595 INFO:tasks.rados.rados.0.smithi071.stdout:361: writing smithi07183189-47 from 3067461 to 3243751 tid 3 2023-12-15T22:57:56.595 INFO:tasks.rados.rados.0.smithi071.stdout:362: snap_create 2023-12-15T22:57:56.595 INFO:tasks.rados.rados.0.smithi071.stdout:360: expect (ObjNum 152 snap 21 seq_num 152) 2023-12-15T22:57:56.693 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T22:57:56.693 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T22:57:56.693 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T22:57:56.749 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:57:56.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:56.802+0000 7f1196f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:56.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:56.808+0000 7f1196f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:56.814 INFO:tasks.rados.rados.0.smithi071.stdout:356: expect (ObjNum 167 snap 26 seq_num 167) 2023-12-15T22:57:56.856 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:56.855+0000 7f19c286d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:56.859 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:56.859+0000 7f19c286d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:56.990 INFO:tasks.rados.rados.0.smithi071.stdout:355: finishing write tid 2 to smithi07183189-24 2023-12-15T22:57:56.990 INFO:tasks.rados.rados.0.smithi071.stdout:358: finishing write tid 1 to smithi07183189-36 2023-12-15T22:57:56.990 INFO:tasks.rados.rados.0.smithi071.stdout:361: finishing write tid 1 to smithi07183189-47 2023-12-15T22:57:56.990 INFO:tasks.rados.rados.0.smithi071.stdout:357: expect (ObjNum 140 snap 17 seq_num 140) 2023-12-15T22:57:56.993 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:57:56.994 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-12-15T22:57:57.006 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":127,"stamp":"2023-12-15T22:57:56.528817+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":102793471,"num_objects":128,"num_object_clones":70,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":811,"num_read_kb":461368,"num_write":777,"num_write_kb":184789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":967,"ondisk_log_size":967,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":336348,"kb_used_data":220732,"kb_used_omap":0,"kb_used_meta":115584,"kb_avail":377151012,"statfs":{"total":386547056640,"available":386202636288,"internally_reserved":0,"allocated":226029568,"data_stored":234312823,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":118358016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3731088,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":17,"num_read_kb":13750,"num_write":8,"num_write_kb":2468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000899"},"pg_stats":[{"pgid":"3.f","version":"73'24","reported_seq":182,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-12-15T22:57:53.259533+0000","last_change":"2023-12-15T22:57:49.325114+0000","last_active":"2023-12-15T22:57:53.259533+0000","last_peered":"2023-12-15T22:57:53.259533+0000","last_clean":"2023-12-15T22:57:53.259533+0000","last_became_active":"2023-12-15T22:56:39.503797+0000","last_became_peered":"2023-12-15T22:56:39.503797+0000","last_unstale":"2023-12-15T22:57:53.259533+0000","last_undegraded":"2023-12-15T22:57:53.259533+0000","last_fullsized":"2023-12-15T22:57:53.259533+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4941225,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":26,"num_read_kb":16107,"num_write":17,"num_write_kb":4759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":164,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-12-15T22:57:54.260036+0000","last_change":"2023-12-15T22:57:54.267006+0000","last_active":"2023-12-15T22:57:54.260036+0000","last_peered":"2023-12-15T22:57:54.260036+0000","last_clean":"2023-12-15T22:57:54.260036+0000","last_became_active":"2023-12-15T22:56:39.552856+0000","last_became_peered":"2023-12-15T22:56:39.552856+0000","last_unstale":"2023-12-15T22:57:54.260036+0000","last_undegraded":"2023-12-15T22:57:54.260036+0000","last_fullsized":"2023-12-15T22:57:54.260036+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"85'112","reported_seq":278,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-12-15T22:57:53.281411+0000","last_change":"2023-12-15T22:57:51.381188+0000","last_active":"2023-12-15T22:57:53.281411+0000","last_peered":"2023-12-15T22:57:53.281411+0000","last_clean":"2023-12-15T22:57:53.281411+0000","last_became_active":"2023-12-15T22:56:39.495490+0000","last_became_peered":"2023-12-15T22:56:39.495490+0000","last_unstale":"2023-12-15T22:57:53.281411+0000","last_undegraded":"2023-12-15T22:57:53.281411+0000","last_fullsized":"2023-12-15T22:57:53.281411+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15159881,"num_objects":16,"num_object_clones":8,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":100,"num_read_kb":65689,"num_write":89,"num_write_kb":25003,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"85'77","reported_seq":235,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-12-15T22:57:53.272252+0000","last_change":"2023-12-15T22:57:49.359987+0000","last_active":"2023-12-15T22:57:53.272252+0000","last_peered":"2023-12-15T22:57:53.272252+0000","last_clean":"2023-12-15T22:57:53.272252+0000","last_became_active":"2023-12-15T22:56:39.579234+0000","last_became_peered":"2023-12-15T22:56:39.579234+0000","last_unstale":"2023-12-15T22:57:53.272252+0000","last_undegraded":"2023-12-15T22:57:53.272252+0000","last_fullsized":"2023-12-15T22:57:53.272252+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9109394,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":37634,"num_write":61,"num_write_kb":16439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"84'35","reported_seq":190,"reported_epoch":89,"state":"active+clean+snaptrim","last_fresh":"2023-12-15T22:57:55.277509+0000","last_change":"2023-12-15T22:57:55.277509+0000","last_active":"2023-12-15T22:57:55.277509+0000","last_peered":"2023-12-15T22:57:55.277509+0000","last_clean":"2023-12-15T22:57:55.277509+0000","last_became_active":"2023-12-15T22:56:39.572176+0000","last_became_peered":"2023-12-15T22:56:39.572176+0000","last_unstale":"2023-12-15T22:57:55.277509+0000","last_undegraded":"2023-12-15T22:57:55.277509+0000","last_fullsized":"2023-12-15T22:57:55.277509+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5815222,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":25,"num_read_kb":8581,"num_write":26,"num_write_kb":7464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"83'50","reported_seq":197,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-12-15T22:57:53.269621+0000","last_change":"2023-12-15T22:57:49.364595+0000","last_active":"2023-12-15T22:57:53.269621+0000","last_peered":"2023-12-15T22:57:53.269621+0000","last_clean":"2023-12-15T22:57:53.269621+0000","last_became_active":"2023-12-15T22:56:39.516361+0000","last_became_peered":"2023-12-15T22:56:39.516361+0000","last_unstale":"2023-12-15T22:57:53.269621+0000","last_undegraded":"2023-12-15T22:57:53.269621+0000","last_fullsized":"2023-12-15T22:57:53.269621+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3109295,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":25,"num_read_kb":19608,"num_write":35,"num_write_kb":10277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"83'180","reported_seq":314,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-12-15T22:57:53.271435+0000","last_change":"2023-12-15T22:57:49.363321+0000","last_active":"2023-12-15T22:57:53.271435+0000","last_peered":"2023-12-15T22:57:53.271435+0000","last_clean":"2023-12-15T22:57:53.271435+0000","last_became_active":"2023-12-15T22:56:39.519748+0000","last_became_peered":"2023-12-15T22:56:39.519748+0000","last_unstale":"2023-12-15T22:57:53.271435+0000","last_undegraded":"2023-12-15T22:57:53.271435+0000","last_fullsized":"2023-12-15T22:57:53.271435+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15092686,"num_objects":27,"num_object_clones":17,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":128,"num_read_kb":74395,"num_write":114,"num_write_kb":32926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"82'46","reported_seq":211,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-12-15T22:57:53.294713+0000","last_change":"2023-12-15T22:57:51.339645+0000","last_active":"2023-12-15T22:57:53.294713+0000","last_peered":"2023-12-15T22:57:53.294713+0000","last_clean":"2023-12-15T22:57:53.294713+0000","last_became_active":"2023-12-15T22:56:39.543179+0000","last_became_peered":"2023-12-15T22:56:39.543179+0000","last_unstale":"2023-12-15T22:57:53.294713+0000","last_undegraded":"2023-12-15T22:57:53.294713+0000","last_fullsized":"2023-12-15T22:57:53.294713+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8857223,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":50,"num_read_kb":40713,"num_write":40,"num_write_kb":10468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":256,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-12-15T22:57:55.279964+0000","last_change":"2023-12-15T22:56:32.607017+0000","last_active":"2023-12-15T22:57:55.279964+0000","last_peered":"2023-12-15T22:57:55.279964+0000","last_clean":"2023-12-15T22:57:55.279964+0000","last_became_active":"2023-12-15T22:56:32.602384+0000","last_became_peered":"2023-12-15T22:56:32.602384+0000","last_unstale":"2023-12-15T22:57:55.279964+0000","last_undegraded":"2023-12-15T22:57:55.279964+0000","last_fullsized":"2023-12-15T22:57:55.279964+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_clean_scrub_stamp":"2023-12-15T22:56:31.525572+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":127,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-12-15T22:57:53.280418+0000","last_change":"2023-12-15T22:56:33.627481+0000","last_active":"2023-12-15T22:57:53.280418+0000","last_peered":"2023-12-15T22:57:53.280418+0000","last_clean":"2023-12-15T22:57:53.280418+0000","last_became_active":"2023-12-15T22:56:33.624689+0000","last_became_peered":"2023-12-15T22:56:33.624689+0000","last_unstale":"2023-12-15T22:57:53.280418+0000","last_undegraded":"2023-12-15T22:57:53.280418+0000","last_fullsized":"2023-12-15T22:57:53.280418+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"69'64","reported_seq":220,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-12-15T22:57:53.281050+0000","last_change":"2023-12-15T22:57:51.337159+0000","last_active":"2023-12-15T22:57:53.281050+0000","last_peered":"2023-12-15T22:57:53.281050+0000","last_clean":"2023-12-15T22:57:53.281050+0000","last_became_active":"2023-12-15T22:56:39.529359+0000","last_became_peered":"2023-12-15T22:56:39.529359+0000","last_unstale":"2023-12-15T22:57:53.281050+0000","last_undegraded":"2023-12-15T22:57:53.281050+0000","last_fullsized":"2023-12-15T22:57:53.281050+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7176589,"num_objects":8,"num_object_clones":3,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":36,"num_read_kb":10704,"num_write":56,"num_write_kb":16150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":129,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-12-15T22:57:53.263846+0000","last_change":"2023-12-15T22:56:33.655797+0000","last_active":"2023-12-15T22:57:53.263846+0000","last_peered":"2023-12-15T22:57:53.263846+0000","last_clean":"2023-12-15T22:57:53.263846+0000","last_became_active":"2023-12-15T22:56:33.652885+0000","last_became_peered":"2023-12-15T22:56:33.652885+0000","last_unstale":"2023-12-15T22:57:53.263846+0000","last_undegraded":"2023-12-15T22:57:53.263846+0000","last_fullsized":"2023-12-15T22:57:53.263846+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"89'62","reported_seq":228,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-12-15T22:57:55.286430+0000","last_change":"2023-12-15T22:57:55.306703+0000","last_active":"2023-12-15T22:57:55.286430+0000","last_peered":"2023-12-15T22:57:55.286430+0000","last_clean":"2023-12-15T22:57:55.286430+0000","last_became_active":"2023-12-15T22:56:39.543509+0000","last_became_peered":"2023-12-15T22:56:39.543509+0000","last_unstale":"2023-12-15T22:57:55.286430+0000","last_undegraded":"2023-12-15T22:57:55.286430+0000","last_fullsized":"2023-12-15T22:57:55.286430+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12353079,"num_objects":11,"num_object_clones":5,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":61,"num_read_kb":43691,"num_write":47,"num_write_kb":13212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":127,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-12-15T22:57:53.265542+0000","last_change":"2023-12-15T22:56:33.600485+0000","last_active":"2023-12-15T22:57:53.265542+0000","last_peered":"2023-12-15T22:57:53.265542+0000","last_clean":"2023-12-15T22:57:53.265542+0000","last_became_active":"2023-12-15T22:56:33.597320+0000","last_became_peered":"2023-12-15T22:56:33.597320+0000","last_unstale":"2023-12-15T22:57:53.265542+0000","last_undegraded":"2023-12-15T22:57:53.265542+0000","last_fullsized":"2023-12-15T22:57:53.265542+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"55'26","reported_seq":185,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-12-15T22:57:55.282250+0000","last_change":"2023-12-15T22:57:55.285464+0000","last_active":"2023-12-15T22:57:55.282250+0000","last_peered":"2023-12-15T22:57:55.282250+0000","last_clean":"2023-12-15T22:57:55.282250+0000","last_became_active":"2023-12-15T22:56:39.560980+0000","last_became_peered":"2023-12-15T22:56:39.560980+0000","last_unstale":"2023-12-15T22:57:55.282250+0000","last_undegraded":"2023-12-15T22:57:55.282250+0000","last_fullsized":"2023-12-15T22:57:55.282250+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5747961,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":13740,"num_write":21,"num_write_kb":6063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":129,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-12-15T22:57:53.258977+0000","last_change":"2023-12-15T22:56:33.648541+0000","last_active":"2023-12-15T22:57:53.258977+0000","last_peered":"2023-12-15T22:57:53.258977+0000","last_clean":"2023-12-15T22:57:53.258977+0000","last_became_active":"2023-12-15T22:56:33.643681+0000","last_became_peered":"2023-12-15T22:56:33.643681+0000","last_unstale":"2023-12-15T22:57:53.258977+0000","last_undegraded":"2023-12-15T22:57:53.258977+0000","last_fullsized":"2023-12-15T22:57:53.258977+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"88'69","reported_seq":236,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-12-15T22:57:54.257624+0000","last_change":"2023-12-15T22:57:54.282425+0000","last_active":"2023-12-15T22:57:54.257624+0000","last_peered":"2023-12-15T22:57:54.257624+0000","last_clean":"2023-12-15T22:57:54.257624+0000","last_became_active":"2023-12-15T22:56:39.506510+0000","last_became_peered":"2023-12-15T22:56:39.506510+0000","last_unstale":"2023-12-15T22:57:54.257624+0000","last_undegraded":"2023-12-15T22:57:54.257624+0000","last_fullsized":"2023-12-15T22:57:54.257624+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2403401,"num_objects":12,"num_object_clones":7,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":66,"num_read_kb":48986,"num_write":44,"num_write_kb":13017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"50'13","reported_seq":163,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-12-15T22:57:53.279361+0000","last_change":"2023-12-15T22:57:51.349413+0000","last_active":"2023-12-15T22:57:53.279361+0000","last_peered":"2023-12-15T22:57:53.279361+0000","last_clean":"2023-12-15T22:57:53.279361+0000","last_became_active":"2023-12-15T22:56:39.511159+0000","last_became_peered":"2023-12-15T22:56:39.511159+0000","last_unstale":"2023-12-15T22:57:53.279361+0000","last_undegraded":"2023-12-15T22:57:53.279361+0000","last_fullsized":"2023-12-15T22:57:53.279361+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":5,"num_read_kb":2,"num_write":9,"num_write_kb":2133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":135,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-12-15T22:57:54.258060+0000","last_change":"2023-12-15T22:56:33.633890+0000","last_active":"2023-12-15T22:57:54.258060+0000","last_peered":"2023-12-15T22:57:54.258060+0000","last_clean":"2023-12-15T22:57:54.258060+0000","last_became_active":"2023-12-15T22:56:33.631134+0000","last_became_peered":"2023-12-15T22:56:33.631134+0000","last_unstale":"2023-12-15T22:57:54.258060+0000","last_undegraded":"2023-12-15T22:57:54.258060+0000","last_fullsized":"2023-12-15T22:57:54.258060+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"54'47","reported_seq":204,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-12-15T22:57:54.254255+0000","last_change":"2023-12-15T22:57:54.267539+0000","last_active":"2023-12-15T22:57:54.254255+0000","last_peered":"2023-12-15T22:57:54.254255+0000","last_clean":"2023-12-15T22:57:54.254255+0000","last_became_active":"2023-12-15T22:56:39.532730+0000","last_became_peered":"2023-12-15T22:56:39.532730+0000","last_unstale":"2023-12-15T22:57:54.254255+0000","last_undegraded":"2023-12-15T22:57:54.254255+0000","last_fullsized":"2023-12-15T22:57:54.254255+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3764735,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":35,"num_read_kb":28298,"num_write":27,"num_write_kb":8830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":127,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-12-15T22:57:55.280394+0000","last_change":"2023-12-15T22:56:33.655736+0000","last_active":"2023-12-15T22:57:55.280394+0000","last_peered":"2023-12-15T22:57:55.280394+0000","last_clean":"2023-12-15T22:57:55.280394+0000","last_became_active":"2023-12-15T22:56:33.652917+0000","last_became_peered":"2023-12-15T22:56:33.652917+0000","last_unstale":"2023-12-15T22:57:55.280394+0000","last_undegraded":"2023-12-15T22:57:55.280394+0000","last_fullsized":"2023-12-15T22:57:55.280394+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"81'56","reported_seq":225,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-12-15T22:57:55.284303+0000","last_change":"2023-12-15T22:57:55.287041+0000","last_active":"2023-12-15T22:57:55.284303+0000","last_peered":"2023-12-15T22:57:55.284303+0000","last_clean":"2023-12-15T22:57:55.284303+0000","last_became_active":"2023-12-15T22:56:39.557252+0000","last_became_peered":"2023-12-15T22:56:39.557252+0000","last_unstale":"2023-12-15T22:57:55.284303+0000","last_undegraded":"2023-12-15T22:57:55.284303+0000","last_fullsized":"2023-12-15T22:57:55.284303+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5916064,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":53,"num_read_kb":39066,"num_write":46,"num_write_kb":13706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":135,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-12-15T22:57:54.256457+0000","last_change":"2023-12-15T22:56:33.632348+0000","last_active":"2023-12-15T22:57:54.256457+0000","last_peered":"2023-12-15T22:57:54.256457+0000","last_clean":"2023-12-15T22:57:54.256457+0000","last_became_active":"2023-12-15T22:56:33.629022+0000","last_became_peered":"2023-12-15T22:56:33.629022+0000","last_unstale":"2023-12-15T22:57:54.256457+0000","last_undegraded":"2023-12-15T22:57:54.256457+0000","last_fullsized":"2023-12-15T22:57:54.256457+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"70'9","reported_seq":178,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-12-15T22:57:54.260902+0000","last_change":"2023-12-15T22:57:54.267213+0000","last_active":"2023-12-15T22:57:54.260902+0000","last_peered":"2023-12-15T22:57:54.260902+0000","last_clean":"2023-12-15T22:57:54.260902+0000","last_became_active":"2023-12-15T22:56:39.535145+0000","last_became_peered":"2023-12-15T22:56:39.535145+0000","last_unstale":"2023-12-15T22:57:54.260902+0000","last_undegraded":"2023-12-15T22:57:54.260902+0000","last_fullsized":"2023-12-15T22:57:54.260902+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2756348,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":14063,"num_write":8,"num_write_kb":2448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":127,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-12-15T22:57:53.270696+0000","last_change":"2023-12-15T22:56:33.627005+0000","last_active":"2023-12-15T22:57:53.270696+0000","last_peered":"2023-12-15T22:57:53.270696+0000","last_clean":"2023-12-15T22:57:53.270696+0000","last_became_active":"2023-12-15T22:56:33.622181+0000","last_became_peered":"2023-12-15T22:56:33.622181+0000","last_unstale":"2023-12-15T22:57:53.270696+0000","last_undegraded":"2023-12-15T22:57:53.270696+0000","last_fullsized":"2023-12-15T22:57:53.270696+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":102203103,"num_objects":126,"num_object_clones":70,"num_object_copies":378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":708,"num_read_kb":461277,"num_write":640,"num_write_kb":182895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":870,"ondisk_log_size":870,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705749,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":88556,"kb_used_data":57572,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94283284,"statfs":{"total":96636764160,"available":96546082816,"internally_reserved":0,"allocated":58953728,"data_stored":60691302,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705748,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":73500,"kb_used_data":46676,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94298340,"statfs":{"total":96636764160,"available":96561500160,"internally_reserved":0,"allocated":47796224,"data_stored":49863938,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705748,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":80192,"kb_used_data":53368,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94291648,"statfs":{"total":96636764160,"available":96554647552,"internally_reserved":0,"allocated":54648832,"data_stored":56224925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705749,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":94100,"kb_used_data":63116,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94277740,"statfs":{"total":96636764160,"available":96540405760,"internally_reserved":0,"allocated":64630784,"data_stored":67532658,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-15T22:57:57.106 INFO:tasks.rados.rados.0.smithi071.stdout:361: finishing write tid 2 to smithi07183189-47 2023-12-15T22:57:57.106 INFO:tasks.rados.rados.0.smithi071.stdout:355: finishing write tid 3 to smithi07183189-24 2023-12-15T22:57:57.106 INFO:tasks.rados.rados.0.smithi071.stdout:358: finishing write tid 2 to smithi07183189-36 2023-12-15T22:57:57.107 INFO:tasks.rados.rados.0.smithi071.stdout:361: finishing write tid 3 to smithi07183189-47 2023-12-15T22:57:57.107 INFO:tasks.rados.rados.0.smithi071.stdout:361: finishing write tid 4 to smithi07183189-47 2023-12-15T22:57:57.107 INFO:tasks.rados.rados.0.smithi071.stdout:361: finishing write tid 5 to smithi07183189-47 2023-12-15T22:57:57.107 INFO:tasks.rados.rados.0.smithi071.stdout:361: oid 47 updating version 0 to 41 2023-12-15T22:57:57.107 INFO:tasks.rados.rados.0.smithi071.stdout:361: oid 47 version 41 is already newer than 41 2023-12-15T22:57:57.107 INFO:tasks.rados.rados.0.smithi071.stdout:361: oid 47 version 41 is already newer than 39 2023-12-15T22:57:57.107 INFO:tasks.rados.rados.0.smithi071.stdout:361: oid 47 updating version 41 to 42 2023-12-15T22:57:57.107 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 42 (ObjNum 193 snap 38 seq_num 193) dirty exists 2023-12-15T22:57:57.107 INFO:tasks.rados.rados.0.smithi071.stdout:361: left oid 47 (ObjNum 193 snap 38 seq_num 193) 2023-12-15T22:57:57.107 INFO:tasks.rados.rados.0.smithi071.stdout:355: finishing write tid 4 to smithi07183189-24 2023-12-15T22:57:57.107 INFO:tasks.rados.rados.0.smithi071.stdout:355: finishing write tid 5 to smithi07183189-24 2023-12-15T22:57:57.107 INFO:tasks.rados.rados.0.smithi071.stdout:355: oid 24 updating version 0 to 16 2023-12-15T22:57:57.107 INFO:tasks.rados.rados.0.smithi071.stdout:355: oid 24 version 16 is already newer than 15 2023-12-15T22:57:57.107 INFO:tasks.rados.rados.0.smithi071.stdout:355: oid 24 version 16 is already newer than 14 2023-12-15T22:57:57.107 INFO:tasks.rados.rados.0.smithi071.stdout:355: oid 24 updating version 16 to 17 2023-12-15T22:57:57.108 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 17 (ObjNum 191 snap 38 seq_num 191) dirty exists 2023-12-15T22:57:57.108 INFO:tasks.rados.rados.0.smithi071.stdout:355: left oid 24 (ObjNum 191 snap 38 seq_num 191) 2023-12-15T22:57:57.108 INFO:tasks.rados.rados.0.smithi071.stdout:358: finishing write tid 3 to smithi07183189-36 2023-12-15T22:57:57.108 INFO:tasks.rados.rados.0.smithi071.stdout:358: finishing write tid 4 to smithi07183189-36 2023-12-15T22:57:57.108 INFO:tasks.rados.rados.0.smithi071.stdout:358: finishing write tid 5 to smithi07183189-36 2023-12-15T22:57:57.108 INFO:tasks.rados.rados.0.smithi071.stdout:358: oid 36 updating version 0 to 116 2023-12-15T22:57:57.108 INFO:tasks.rados.rados.0.smithi071.stdout:358: oid 36 version 116 is already newer than 115 2023-12-15T22:57:57.108 INFO:tasks.rados.rados.0.smithi071.stdout:358: oid 36 updating version 116 to 118 2023-12-15T22:57:57.108 INFO:tasks.rados.rados.0.smithi071.stdout:358: oid 36 version 118 is already newer than 117 2023-12-15T22:57:57.108 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 118 (ObjNum 192 snap 38 seq_num 192) dirty exists 2023-12-15T22:57:57.108 INFO:tasks.rados.rados.0.smithi071.stdout:358: left oid 36 (ObjNum 192 snap 38 seq_num 192) 2023-12-15T22:57:57.108 INFO:tasks.rados.rados.0.smithi071.stdout:359: expect (ObjNum 117 snap 10 seq_num 117) 2023-12-15T22:57:57.327 INFO:tasks.rados.rados.0.smithi071.stdout:355: done (7 left) 2023-12-15T22:57:57.327 INFO:tasks.rados.rados.0.smithi071.stdout:356: done (6 left) 2023-12-15T22:57:57.327 INFO:tasks.rados.rados.0.smithi071.stdout:357: done (5 left) 2023-12-15T22:57:57.327 INFO:tasks.rados.rados.0.smithi071.stdout:358: done (4 left) 2023-12-15T22:57:57.327 INFO:tasks.rados.rados.0.smithi071.stdout:359: done (3 left) 2023-12-15T22:57:57.327 INFO:tasks.rados.rados.0.smithi071.stdout:360: done (2 left) 2023-12-15T22:57:57.328 INFO:tasks.rados.rados.0.smithi071.stdout:361: done (1 left) 2023-12-15T22:57:57.328 INFO:tasks.rados.rados.0.smithi071.stdout:362: done (0 left) 2023-12-15T22:57:57.328 INFO:tasks.rados.rados.0.smithi071.stdout:363: snap_remove snap 34 2023-12-15T22:57:58.270 INFO:tasks.rados.rados.0.smithi071.stdout:363: done (0 left) 2023-12-15T22:57:58.270 INFO:tasks.rados.rados.0.smithi071.stdout:364: snap_create 2023-12-15T22:57:58.940 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:57:59.049 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:59.048+0000 7f0339c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:59.052 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:57:59.052+0000 7f0339c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:57:59.271 INFO:tasks.rados.rados.0.smithi071.stdout:364: done (0 left) 2023-12-15T22:57:59.272 INFO:tasks.rados.rados.0.smithi071.stdout:365: write (excl) oid 4 current snap is 40 2023-12-15T22:57:59.272 INFO:tasks.rados.rados.0.smithi071.stdout:365: seq_num 194 ranges {460558=707131,1869389=508716,3094884=165674} 2023-12-15T22:57:59.278 INFO:tasks.rados.rados.0.smithi071.stdout:365: writing smithi07183189-4 from 460558 to 1167689 tid 1 2023-12-15T22:57:59.286 INFO:tasks.rados.rados.0.smithi071.stdout:365: writing smithi07183189-4 from 1869389 to 2378105 tid 2 2023-12-15T22:57:59.291 INFO:tasks.rados.rados.0.smithi071.stdout:365: writing smithi07183189-4 from 3094884 to 3260558 tid 3 2023-12-15T22:57:59.291 INFO:tasks.rados.rados.0.smithi071.stdout:366: write (excl) oid 22 current snap is 40 2023-12-15T22:57:59.291 INFO:tasks.rados.rados.0.smithi071.stdout:366: seq_num 195 ranges {477365=782380,1927871=768727,3122307=155058} 2023-12-15T22:57:59.300 INFO:tasks.rados.rados.0.smithi071.stdout:366: writing smithi07183189-22 from 477365 to 1259745 tid 1 2023-12-15T22:57:59.310 INFO:tasks.rados.rados.0.smithi071.stdout:366: writing smithi07183189-22 from 1927871 to 2696598 tid 2 2023-12-15T22:57:59.314 INFO:tasks.rados.rados.0.smithi071.stdout:366: writing smithi07183189-22 from 3122307 to 3277365 tid 3 2023-12-15T22:57:59.315 INFO:tasks.rados.rados.0.smithi071.stdout:367: write oid 34 current snap is 40 2023-12-15T22:57:59.315 INFO:tasks.rados.rados.0.smithi071.stdout:367: seq_num 196 ranges {494172=457629,1586353=512385,2749730=544442} 2023-12-15T22:57:59.320 INFO:tasks.rados.rados.0.smithi071.stdout:367: writing smithi07183189-34 from 494172 to 951801 tid 1 2023-12-15T22:57:59.326 INFO:tasks.rados.rados.0.smithi071.stdout:367: writing smithi07183189-34 from 1586353 to 2098738 tid 2 2023-12-15T22:57:59.333 INFO:tasks.rados.rados.0.smithi071.stdout:367: writing smithi07183189-34 from 2749730 to 3294172 tid 3 2023-12-15T22:57:59.333 INFO:tasks.rados.rados.0.smithi071.stdout:368: write oid 15 current snap is 40 2023-12-15T22:57:59.333 INFO:tasks.rados.rados.0.smithi071.stdout:368: seq_num 197 ranges {510979=532878,1528482=772396,3060800=250179} 2023-12-15T22:57:59.339 INFO:tasks.rados.rados.0.smithi071.stdout:368: writing smithi07183189-15 from 510979 to 1043857 tid 1 2023-12-15T22:57:59.346 INFO:tasks.rados.rados.0.smithi071.stdout:368: writing smithi07183189-15 from 1528482 to 2300878 tid 2 2023-12-15T22:57:59.352 INFO:tasks.rados.rados.0.smithi071.stdout:368: writing smithi07183189-15 from 3060800 to 3310979 tid 3 2023-12-15T22:57:59.352 INFO:tasks.rados.rados.0.smithi071.stdout:369: read oid 10 snap 38 2023-12-15T22:57:59.352 INFO:tasks.rados.rados.0.smithi071.stdout:369: expect (ObjNum 147 snap 20 seq_num 147) 2023-12-15T22:57:59.352 INFO:tasks.rados.rados.0.smithi071.stdout:370: write (excl) oid 12 current snap is 40 2023-12-15T22:57:59.353 INFO:tasks.rados.rados.0.smithi071.stdout:370: seq_num 198 ranges {527786=724480,1703317=516054,2804576=435035} 2023-12-15T22:57:59.359 INFO:tasks.rados.rados.0.smithi071.stdout:370: writing smithi07183189-12 from 527786 to 1252266 tid 1 2023-12-15T22:57:59.364 INFO:tasks.rados.rados.0.smithi071.stdout:370: writing smithi07183189-12 from 1703317 to 2219371 tid 2 2023-12-15T22:57:59.370 INFO:tasks.rados.rados.0.smithi071.stdout:370: writing smithi07183189-12 from 2804576 to 3239611 tid 3 2023-12-15T22:57:59.371 INFO:tasks.rados.rados.0.smithi071.stdout:371: read oid 37 snap 38 2023-12-15T22:57:59.371 INFO:tasks.rados.rados.0.smithi071.stdout:371: expect deleted 2023-12-15T22:57:59.371 INFO:tasks.rados.rados.0.smithi071.stdout:372: write (excl) oid 19 current snap is 40 2023-12-15T22:57:59.371 INFO:tasks.rados.rados.0.smithi071.stdout:365: finishing write tid 1 to smithi07183189-4 2023-12-15T22:57:59.371 INFO:tasks.rados.rados.0.smithi071.stdout:372: seq_num 199 ranges {544593=799729,1761799=776065,2948352=396241} 2023-12-15T22:57:59.378 INFO:tasks.rados.rados.0.smithi071.stdout:372: writing smithi07183189-19 from 544593 to 1344322 tid 1 2023-12-15T22:57:59.385 INFO:tasks.rados.rados.0.smithi071.stdout:372: writing smithi07183189-19 from 1761799 to 2537864 tid 2 2023-12-15T22:57:59.390 INFO:tasks.rados.rados.0.smithi071.stdout:372: writing smithi07183189-19 from 2948352 to 3344593 tid 3 2023-12-15T22:57:59.390 INFO:tasks.rados.rados.0.smithi071.stdout:373: snap_create 2023-12-15T22:57:59.390 INFO:tasks.rados.rados.0.smithi071.stdout:366: finishing write tid 1 to smithi07183189-22 2023-12-15T22:57:59.390 INFO:tasks.rados.rados.0.smithi071.stdout:365: finishing write tid 2 to smithi07183189-4 2023-12-15T22:57:59.390 INFO:tasks.rados.rados.0.smithi071.stdout:368: finishing write tid 1 to smithi07183189-15 2023-12-15T22:57:59.406 INFO:tasks.rados.rados.0.smithi071.stdout:366: finishing write tid 2 to smithi07183189-22 2023-12-15T22:57:59.421 INFO:tasks.rados.rados.0.smithi071.stdout:365: finishing write tid 3 to smithi07183189-4 2023-12-15T22:57:59.460 INFO:tasks.rados.rados.0.smithi071.stdout:367: finishing write tid 1 to smithi07183189-34 2023-12-15T22:57:59.480 INFO:tasks.rados.rados.0.smithi071.stdout:366: finishing write tid 3 to smithi07183189-22 2023-12-15T22:57:59.483 INFO:tasks.rados.rados.0.smithi071.stdout:368: finishing write tid 2 to smithi07183189-15 2023-12-15T22:57:59.485 INFO:tasks.rados.rados.0.smithi071.stdout:372: finishing write tid 1 to smithi07183189-19 2023-12-15T22:57:59.486 INFO:tasks.rados.rados.0.smithi071.stdout:370: finishing write tid 1 to smithi07183189-12 2023-12-15T22:57:59.488 INFO:tasks.rados.rados.0.smithi071.stdout:366: finishing write tid 4 to smithi07183189-22 2023-12-15T22:57:59.490 INFO:tasks.rados.rados.0.smithi071.stdout:366: finishing write tid 5 to smithi07183189-22 2023-12-15T22:57:59.490 INFO:tasks.rados.rados.0.smithi071.stdout:366: oid 22 updating version 0 to 77 2023-12-15T22:57:59.490 INFO:tasks.rados.rados.0.smithi071.stdout:366: oid 22 version 77 is already newer than 74 2023-12-15T22:57:59.490 INFO:tasks.rados.rados.0.smithi071.stdout:366: oid 22 version 77 is already newer than 77 2023-12-15T22:57:59.490 INFO:tasks.rados.rados.0.smithi071.stdout:366: oid 22 version 77 is already newer than 75 2023-12-15T22:57:59.491 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 77 (ObjNum 195 snap 40 seq_num 195) dirty exists 2023-12-15T22:57:59.491 INFO:tasks.rados.rados.0.smithi071.stdout:366: left oid 22 (ObjNum 195 snap 40 seq_num 195) 2023-12-15T22:57:59.491 INFO:tasks.rados.rados.0.smithi071.stdout:365: finishing write tid 4 to smithi07183189-4 2023-12-15T22:57:59.495 INFO:tasks.rados.rados.0.smithi071.stdout:365: finishing write tid 5 to smithi07183189-4 2023-12-15T22:57:59.495 INFO:tasks.rados.rados.0.smithi071.stdout:365: oid 4 updating version 0 to 121 2023-12-15T22:57:59.495 INFO:tasks.rados.rados.0.smithi071.stdout:365: oid 4 version 121 is already newer than 120 2023-12-15T22:57:59.495 INFO:tasks.rados.rados.0.smithi071.stdout:365: oid 4 updating version 121 to 122 2023-12-15T22:57:59.495 INFO:tasks.rados.rados.0.smithi071.stdout:365: oid 4 updating version 122 to 123 2023-12-15T22:57:59.495 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 4 v 123 (ObjNum 194 snap 40 seq_num 194) dirty exists 2023-12-15T22:57:59.495 INFO:tasks.rados.rados.0.smithi071.stdout:365: left oid 4 (ObjNum 194 snap 40 seq_num 194) 2023-12-15T22:57:59.512 INFO:tasks.rados.rados.0.smithi071.stdout:367: finishing write tid 2 to smithi07183189-34 2023-12-15T22:57:59.522 INFO:tasks.rados.rados.0.smithi071.stdout:368: finishing write tid 3 to smithi07183189-15 2023-12-15T22:57:59.531 INFO:tasks.rados.rados.0.smithi071.stdout:370: finishing write tid 2 to smithi07183189-12 2023-12-15T22:57:59.531 INFO:tasks.rados.rados.0.smithi071.stdout:372: finishing write tid 2 to smithi07183189-19 2023-12-15T22:57:59.550 INFO:tasks.rados.rados.0.smithi071.stdout:368: finishing write tid 4 to smithi07183189-15 2023-12-15T22:57:59.551 INFO:tasks.rados.rados.0.smithi071.stdout:367: finishing write tid 3 to smithi07183189-34 2023-12-15T22:57:59.553 INFO:tasks.rados.rados.0.smithi071.stdout:367: finishing write tid 4 to smithi07183189-34 2023-12-15T22:57:59.554 INFO:tasks.rados.rados.0.smithi071.stdout:368: finishing write tid 5 to smithi07183189-15 2023-12-15T22:57:59.554 INFO:tasks.rados.rados.0.smithi071.stdout:368: oid 15 updating version 0 to 71 2023-12-15T22:57:59.554 INFO:tasks.rados.rados.0.smithi071.stdout:368: oid 15 version 71 is already newer than 69 2023-12-15T22:57:59.554 INFO:tasks.rados.rados.0.smithi071.stdout:368: oid 15 updating version 71 to 72 2023-12-15T22:57:59.554 INFO:tasks.rados.rados.0.smithi071.stdout:368: oid 15 version 72 is already newer than 70 2023-12-15T22:57:59.554 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 72 (ObjNum 197 snap 40 seq_num 197) dirty exists 2023-12-15T22:57:59.554 INFO:tasks.rados.rados.0.smithi071.stdout:368: left oid 15 (ObjNum 197 snap 40 seq_num 197) 2023-12-15T22:57:59.555 INFO:tasks.rados.rados.0.smithi071.stdout:367: finishing write tid 5 to smithi07183189-34 2023-12-15T22:57:59.555 INFO:tasks.rados.rados.0.smithi071.stdout:367: oid 34 updating version 0 to 14 2023-12-15T22:57:59.555 INFO:tasks.rados.rados.0.smithi071.stdout:367: oid 34 version 14 is already newer than 12 2023-12-15T22:57:59.555 INFO:tasks.rados.rados.0.smithi071.stdout:367: oid 34 version 14 is already newer than 14 2023-12-15T22:57:59.555 INFO:tasks.rados.rados.0.smithi071.stdout:367: oid 34 version 14 is already newer than 11 2023-12-15T22:57:59.555 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 14 (ObjNum 196 snap 40 seq_num 196) dirty exists 2023-12-15T22:57:59.555 INFO:tasks.rados.rados.0.smithi071.stdout:367: left oid 34 (ObjNum 196 snap 40 seq_num 196) 2023-12-15T22:57:59.564 INFO:tasks.rados.rados.0.smithi071.stdout:369: expect (ObjNum 147 snap 20 seq_num 147) 2023-12-15T22:57:59.671 INFO:tasks.rados.rados.0.smithi071.stdout:370: finishing write tid 3 to smithi07183189-12 2023-12-15T22:57:59.671 INFO:tasks.rados.rados.0.smithi071.stdout:372: finishing write tid 3 to smithi07183189-19 2023-12-15T22:57:59.671 INFO:tasks.rados.rados.0.smithi071.stdout:372: finishing write tid 4 to smithi07183189-19 2023-12-15T22:57:59.671 INFO:tasks.rados.rados.0.smithi071.stdout:370: finishing write tid 4 to smithi07183189-12 2023-12-15T22:57:59.671 INFO:tasks.rados.rados.0.smithi071.stdout:372: finishing write tid 5 to smithi07183189-19 2023-12-15T22:57:59.672 INFO:tasks.rados.rados.0.smithi071.stdout:372: oid 19 updating version 0 to 87 2023-12-15T22:57:59.672 INFO:tasks.rados.rados.0.smithi071.stdout:372: oid 19 updating version 87 to 88 2023-12-15T22:57:59.672 INFO:tasks.rados.rados.0.smithi071.stdout:372: oid 19 version 88 is already newer than 85 2023-12-15T22:57:59.672 INFO:tasks.rados.rados.0.smithi071.stdout:372: oid 19 version 88 is already newer than 86 2023-12-15T22:57:59.672 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 88 (ObjNum 199 snap 40 seq_num 199) dirty exists 2023-12-15T22:57:59.672 INFO:tasks.rados.rados.0.smithi071.stdout:372: left oid 19 (ObjNum 199 snap 40 seq_num 199) 2023-12-15T22:57:59.672 INFO:tasks.rados.rados.0.smithi071.stdout:370: finishing write tid 5 to smithi07183189-12 2023-12-15T22:57:59.672 INFO:tasks.rados.rados.0.smithi071.stdout:370: oid 12 updating version 0 to 46 2023-12-15T22:57:59.672 INFO:tasks.rados.rados.0.smithi071.stdout:370: oid 12 version 46 is already newer than 45 2023-12-15T22:57:59.672 INFO:tasks.rados.rados.0.smithi071.stdout:370: oid 12 updating version 46 to 48 2023-12-15T22:57:59.672 INFO:tasks.rados.rados.0.smithi071.stdout:370: oid 12 version 48 is already newer than 47 2023-12-15T22:57:59.672 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 12 v 48 (ObjNum 198 snap 40 seq_num 198) dirty exists 2023-12-15T22:57:59.672 INFO:tasks.rados.rados.0.smithi071.stdout:370: left oid 12 (ObjNum 198 snap 40 seq_num 198) 2023-12-15T22:58:00.278 INFO:tasks.rados.rados.0.smithi071.stdout:365: done (8 left) 2023-12-15T22:58:00.278 INFO:tasks.rados.rados.0.smithi071.stdout:366: done (7 left) 2023-12-15T22:58:00.279 INFO:tasks.rados.rados.0.smithi071.stdout:367: done (6 left) 2023-12-15T22:58:00.279 INFO:tasks.rados.rados.0.smithi071.stdout:368: done (5 left) 2023-12-15T22:58:00.279 INFO:tasks.rados.rados.0.smithi071.stdout:369: done (4 left) 2023-12-15T22:58:00.279 INFO:tasks.rados.rados.0.smithi071.stdout:370: done (3 left) 2023-12-15T22:58:00.279 INFO:tasks.rados.rados.0.smithi071.stdout:371: done (2 left) 2023-12-15T22:58:00.279 INFO:tasks.rados.rados.0.smithi071.stdout:372: done (1 left) 2023-12-15T22:58:00.279 INFO:tasks.rados.rados.0.smithi071.stdout:373: done (0 left) 2023-12-15T22:58:00.279 INFO:tasks.rados.rados.0.smithi071.stdout:374: write oid 33 current snap is 41 2023-12-15T22:58:00.279 INFO:tasks.rados.rados.0.smithi071.stdout:374: seq_num 200 ranges {561400=474978,1820281=519723,2859422=457579} 2023-12-15T22:58:00.285 INFO:tasks.rados.rados.0.smithi071.stdout:374: writing smithi07183189-33 from 561400 to 1036378 tid 1 2023-12-15T22:58:00.294 INFO:tasks.rados.rados.0.smithi071.stdout:374: writing smithi07183189-33 from 1820281 to 2340004 tid 2 2023-12-15T22:58:00.301 INFO:tasks.rados.rados.0.smithi071.stdout:374: writing smithi07183189-33 from 2859422 to 3317001 tid 3 2023-12-15T22:58:00.301 INFO:tasks.rados.rados.0.smithi071.stdout:375: read oid 5 snap -1 2023-12-15T22:58:00.301 INFO:tasks.rados.rados.0.smithi071.stdout:375: expect (ObjNum 126 snap 15 seq_num 126) 2023-12-15T22:58:00.301 INFO:tasks.rados.rados.0.smithi071.stdout:376: read oid 32 snap -1 2023-12-15T22:58:00.301 INFO:tasks.rados.rados.0.smithi071.stdout:376: expect (ObjNum 174 snap 29 seq_num 174) 2023-12-15T22:58:00.301 INFO:tasks.rados.rados.0.smithi071.stdout:377: read oid 38 snap -1 2023-12-15T22:58:00.301 INFO:tasks.rados.rados.0.smithi071.stdout:377: expect (ObjNum 54 snap 0 seq_num 54) 2023-12-15T22:58:00.301 INFO:tasks.rados.rados.0.smithi071.stdout:378: snap_create 2023-12-15T22:58:00.350 INFO:tasks.rados.rados.0.smithi071.stdout:377: expect (ObjNum 54 snap 0 seq_num 54) 2023-12-15T22:58:00.512 INFO:tasks.rados.rados.0.smithi071.stdout:376: expect (ObjNum 174 snap 29 seq_num 174) 2023-12-15T22:58:00.629 INFO:tasks.rados.rados.0.smithi071.stdout:374: finishing write tid 1 to smithi07183189-33 2023-12-15T22:58:00.629 INFO:tasks.rados.rados.0.smithi071.stdout:374: finishing write tid 2 to smithi07183189-33 2023-12-15T22:58:00.629 INFO:tasks.rados.rados.0.smithi071.stdout:374: finishing write tid 3 to smithi07183189-33 2023-12-15T22:58:00.629 INFO:tasks.rados.rados.0.smithi071.stdout:374: finishing write tid 4 to smithi07183189-33 2023-12-15T22:58:00.629 INFO:tasks.rados.rados.0.smithi071.stdout:374: finishing write tid 5 to smithi07183189-33 2023-12-15T22:58:00.629 INFO:tasks.rados.rados.0.smithi071.stdout:374: oid 33 updating version 0 to 187 2023-12-15T22:58:00.630 INFO:tasks.rados.rados.0.smithi071.stdout:374: oid 33 updating version 187 to 188 2023-12-15T22:58:00.630 INFO:tasks.rados.rados.0.smithi071.stdout:374: oid 33 version 188 is already newer than 186 2023-12-15T22:58:00.630 INFO:tasks.rados.rados.0.smithi071.stdout:374: oid 33 version 188 is already newer than 185 2023-12-15T22:58:00.630 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 33 v 188 (ObjNum 200 snap 41 seq_num 200) dirty exists 2023-12-15T22:58:00.630 INFO:tasks.rados.rados.0.smithi071.stdout:374: left oid 33 (ObjNum 200 snap 41 seq_num 200) 2023-12-15T22:58:00.630 INFO:tasks.rados.rados.0.smithi071.stdout:375: expect (ObjNum 126 snap 15 seq_num 126) 2023-12-15T22:58:01.123 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:58:01.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:01.226+0000 7fcb7959e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:01.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:01.230+0000 7fcb7959e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:01.282 INFO:tasks.rados.rados.0.smithi071.stdout:374: done (4 left) 2023-12-15T22:58:01.282 INFO:tasks.rados.rados.0.smithi071.stdout:375: done (3 left) 2023-12-15T22:58:01.282 INFO:tasks.rados.rados.0.smithi071.stdout:376: done (2 left) 2023-12-15T22:58:01.282 INFO:tasks.rados.rados.0.smithi071.stdout:377: done (1 left) 2023-12-15T22:58:01.282 INFO:tasks.rados.rados.0.smithi071.stdout:378: done (0 left) 2023-12-15T22:58:01.282 INFO:tasks.rados.rados.0.smithi071.stdout:379: delete oid 8 current snap is 42 2023-12-15T22:58:01.312 INFO:tasks.rados.rados.0.smithi071.stdout:379: done (0 left) 2023-12-15T22:58:01.312 INFO:tasks.rados.rados.0.smithi071.stdout:380: read oid 14 snap -1 2023-12-15T22:58:01.312 INFO:tasks.rados.rados.0.smithi071.stdout:380: expect (ObjNum 140 snap 17 seq_num 140) 2023-12-15T22:58:01.312 INFO:tasks.rados.rados.0.smithi071.stdout:381: snap_create 2023-12-15T22:58:01.325 INFO:tasks.rados.rados.0.smithi071.stdout:380: expect (ObjNum 140 snap 17 seq_num 140) 2023-12-15T22:58:02.009 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T22:58:02.009 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T22:58:02.009 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-12-15T22:58:02.010 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-12-15T22:58:02.118 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:02.113+0000 7fd8bee6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:02.120 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:02.121+0000 7fd8bee6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:02.133 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:58:02.133 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:58:02.133 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:58:02.133 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:58:02.133 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:58:02.133 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:58:02.133 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:58:02.133 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:58:02.133 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:58:02.133 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:58:02.133 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:58:02.133 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:58:02.136 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:58:02.136 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:58:02.136 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-12-15T22:58:02.240 INFO:tasks.rados.rados.0.smithi071.stdout:380: done (1 left) 2023-12-15T22:58:02.240 INFO:tasks.rados.rados.0.smithi071.stdout:381: done (0 left) 2023-12-15T22:58:02.240 INFO:tasks.rados.rados.0.smithi071.stdout:382: snap_remove snap 40 2023-12-15T22:58:02.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:02.239+0000 7f6c8806d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:02.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:02.246+0000 7f6c8806d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:02.258 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:58:02.258 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:58:02.258 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:58:02.258 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:58:02.258 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:58:02.259 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:58:02.259 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:58:02.259 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:58:02.259 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:58:02.259 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:58:02.259 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:58:02.259 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:58:02.261 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:58:02.261 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:58:02.262 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-12-15T22:58:02.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:02.363+0000 7f8759f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:02.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:02.370+0000 7f8759f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:02.383 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:58:02.383 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:58:02.383 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:58:02.383 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:58:02.383 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:58:02.383 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:58:02.383 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:58:02.383 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:58:02.383 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:58:02.384 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:58:02.384 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:58:02.384 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:58:02.386 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:58:02.386 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:58:02.386 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-12-15T22:58:02.492 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:02.487+0000 7fc84db1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:02.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:02.495+0000 7fc84db1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:02.507 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:58:02.508 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:58:02.508 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:58:02.508 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:58:02.508 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:58:02.508 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:58:02.508 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:58:02.508 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:58:02.508 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:58:02.508 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:58:02.508 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:58:02.508 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:58:02.510 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:58:02.511 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:58:03.244 INFO:tasks.rados.rados.0.smithi071.stdout:382: done (0 left) 2023-12-15T22:58:03.244 INFO:tasks.rados.rados.0.smithi071.stdout:383: write (excl) oid 17 current snap is 43 2023-12-15T22:58:03.244 INFO:tasks.rados.rados.0.smithi071.stdout:383: seq_num 202 ranges {595014=625476,1820892=523392,2797915=596476} 2023-12-15T22:58:03.250 INFO:tasks.rados.rados.0.smithi071.stdout:383: writing smithi07183189-17 from 595014 to 1220490 tid 1 2023-12-15T22:58:03.256 INFO:tasks.rados.rados.0.smithi071.stdout:383: writing smithi07183189-17 from 1820892 to 2344284 tid 2 2023-12-15T22:58:03.263 INFO:tasks.rados.rados.0.smithi071.stdout:383: writing smithi07183189-17 from 2797915 to 3394391 tid 3 2023-12-15T22:58:03.263 INFO:tasks.rados.rados.0.smithi071.stdout:384: read oid 20 snap -1 2023-12-15T22:58:03.263 INFO:tasks.rados.rados.0.smithi071.stdout:384: expect (ObjNum 154 snap 21 seq_num 154) 2023-12-15T22:58:03.263 INFO:tasks.rados.rados.0.smithi071.stdout:385: delete oid 42 current snap is 43 2023-12-15T22:58:03.318 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:58:03.319 INFO:tasks.rados.rados.0.smithi071.stdout:384: expect (ObjNum 154 snap 21 seq_num 154) 2023-12-15T22:58:03.432 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:03.431+0000 7f8075a64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:03.435 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:03.435+0000 7f8075a64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:03.472 INFO:tasks.rados.rados.0.smithi071.stdout:384: done (2 left) 2023-12-15T22:58:03.472 INFO:tasks.rados.rados.0.smithi071.stdout:385: done (1 left) 2023-12-15T22:58:03.472 INFO:tasks.rados.rados.0.smithi071.stdout:386: read oid 36 snap 41 2023-12-15T22:58:03.472 INFO:tasks.rados.rados.0.smithi071.stdout:386: expect (ObjNum 192 snap 38 seq_num 192) 2023-12-15T22:58:03.472 INFO:tasks.rados.rados.0.smithi071.stdout:387: read oid 13 snap -1 2023-12-15T22:58:03.472 INFO:tasks.rados.rados.0.smithi071.stdout:387: expect (ObjNum 189 snap 37 seq_num 189) 2023-12-15T22:58:03.473 INFO:tasks.rados.rados.0.smithi071.stdout:388: read oid 19 snap -1 2023-12-15T22:58:03.473 INFO:tasks.rados.rados.0.smithi071.stdout:388: expect (ObjNum 199 snap 40 seq_num 199) 2023-12-15T22:58:03.473 INFO:tasks.rados.rados.0.smithi071.stdout:389: read oid 38 snap -1 2023-12-15T22:58:03.473 INFO:tasks.rados.rados.0.smithi071.stdout:389: expect (ObjNum 54 snap 0 seq_num 54) 2023-12-15T22:58:03.473 INFO:tasks.rados.rados.0.smithi071.stdout:390: read oid 9 snap -1 2023-12-15T22:58:03.473 INFO:tasks.rados.rados.0.smithi071.stdout:390: expect (ObjNum 170 snap 26 seq_num 170) 2023-12-15T22:58:03.473 INFO:tasks.rados.rados.0.smithi071.stdout:391: snap_create 2023-12-15T22:58:03.473 INFO:tasks.rados.rados.0.smithi071.stdout:383: finishing write tid 1 to smithi07183189-17 2023-12-15T22:58:03.473 INFO:tasks.rados.rados.0.smithi071.stdout:383: finishing write tid 2 to smithi07183189-17 2023-12-15T22:58:03.473 INFO:tasks.rados.rados.0.smithi071.stdout:383: finishing write tid 3 to smithi07183189-17 2023-12-15T22:58:03.473 INFO:tasks.rados.rados.0.smithi071.stdout:383: finishing write tid 4 to smithi07183189-17 2023-12-15T22:58:03.473 INFO:tasks.rados.rados.0.smithi071.stdout:383: finishing write tid 5 to smithi07183189-17 2023-12-15T22:58:03.473 INFO:tasks.rados.rados.0.smithi071.stdout:383: oid 17 updating version 0 to 127 2023-12-15T22:58:03.473 INFO:tasks.rados.rados.0.smithi071.stdout:383: oid 17 version 127 is already newer than 125 2023-12-15T22:58:03.474 INFO:tasks.rados.rados.0.smithi071.stdout:383: oid 17 version 127 is already newer than 127 2023-12-15T22:58:03.474 INFO:tasks.rados.rados.0.smithi071.stdout:383: oid 17 updating version 127 to 128 2023-12-15T22:58:03.474 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 17 v 128 (ObjNum 202 snap 43 seq_num 202) dirty exists 2023-12-15T22:58:03.474 INFO:tasks.rados.rados.0.smithi071.stdout:383: left oid 17 (ObjNum 202 snap 43 seq_num 202) 2023-12-15T22:58:03.498 INFO:tasks.rados.rados.0.smithi071.stdout:388: expect (ObjNum 199 snap 40 seq_num 199) 2023-12-15T22:58:03.671 INFO:tasks.rados.rados.0.smithi071.stdout:390: expect (ObjNum 170 snap 26 seq_num 170) 2023-12-15T22:58:03.858 INFO:tasks.rados.rados.0.smithi071.stdout:387: expect (ObjNum 189 snap 37 seq_num 189) 2023-12-15T22:58:04.027 INFO:tasks.rados.rados.0.smithi071.stdout:389: expect (ObjNum 54 snap 0 seq_num 54) 2023-12-15T22:58:04.178 INFO:tasks.rados.rados.0.smithi071.stdout:386: expect (ObjNum 192 snap 38 seq_num 192) 2023-12-15T22:58:04.305 INFO:tasks.rados.rados.0.smithi071.stdout:383: done (6 left) 2023-12-15T22:58:04.305 INFO:tasks.rados.rados.0.smithi071.stdout:386: done (5 left) 2023-12-15T22:58:04.305 INFO:tasks.rados.rados.0.smithi071.stdout:387: done (4 left) 2023-12-15T22:58:04.305 INFO:tasks.rados.rados.0.smithi071.stdout:388: done (3 left) 2023-12-15T22:58:04.305 INFO:tasks.rados.rados.0.smithi071.stdout:389: done (2 left) 2023-12-15T22:58:04.305 INFO:tasks.rados.rados.0.smithi071.stdout:390: done (1 left) 2023-12-15T22:58:04.305 INFO:tasks.rados.rados.0.smithi071.stdout:391: done (0 left) 2023-12-15T22:58:04.305 INFO:tasks.rados.rados.0.smithi071.stdout:392: read oid 26 snap 37 2023-12-15T22:58:04.305 INFO:tasks.rados.rados.0.smithi071.stdout:392: expect (ObjNum 181 snap 33 seq_num 181) 2023-12-15T22:58:04.305 INFO:tasks.rados.rados.0.smithi071.stdout:393: delete oid 29 current snap is 44 2023-12-15T22:58:04.322 INFO:tasks.rados.rados.0.smithi071.stdout:392: expect (ObjNum 181 snap 33 seq_num 181) 2023-12-15T22:58:04.447 INFO:tasks.rados.rados.0.smithi071.stdout:392: done (1 left) 2023-12-15T22:58:04.447 INFO:tasks.rados.rados.0.smithi071.stdout:393: done (0 left) 2023-12-15T22:58:04.447 INFO:tasks.rados.rados.0.smithi071.stdout:394: write oid 4 current snap is 44 2023-12-15T22:58:04.447 INFO:tasks.rados.rados.0.smithi071.stdout:394: seq_num 205 ranges {645435=451223,1879985=787072,3280184=165251} 2023-12-15T22:58:04.452 INFO:tasks.rados.rados.0.smithi071.stdout:394: writing smithi07183189-4 from 645435 to 1096658 tid 1 2023-12-15T22:58:04.463 INFO:tasks.rados.rados.0.smithi071.stdout:394: writing smithi07183189-4 from 1879985 to 2667057 tid 2 2023-12-15T22:58:04.468 INFO:tasks.rados.rados.0.smithi071.stdout:394: writing smithi07183189-4 from 3280184 to 3445435 tid 3 2023-12-15T22:58:04.468 INFO:tasks.rados.rados.0.smithi071.stdout:395: read oid 29 snap -1 2023-12-15T22:58:04.468 INFO:tasks.rados.rados.0.smithi071.stdout:395: expect deleted 2023-12-15T22:58:04.469 INFO:tasks.rados.rados.0.smithi071.stdout:396: read oid 44 snap -1 2023-12-15T22:58:04.469 INFO:tasks.rados.rados.0.smithi071.stdout:396: expect (ObjNum 105 snap 9 seq_num 105) 2023-12-15T22:58:04.469 INFO:tasks.rados.rados.0.smithi071.stdout:397: write (excl) oid 23 current snap is 44 2023-12-15T22:58:04.469 INFO:tasks.rados.rados.0.smithi071.stdout:397: seq_num 206 ranges {662242=642825,2054820=530730,3307607=154635} 2023-12-15T22:58:04.476 INFO:tasks.rados.rados.0.smithi071.stdout:397: writing smithi07183189-23 from 662242 to 1305067 tid 1 2023-12-15T22:58:04.483 INFO:tasks.rados.rados.0.smithi071.stdout:397: writing smithi07183189-23 from 2054820 to 2585550 tid 2 2023-12-15T22:58:04.488 INFO:tasks.rados.rados.0.smithi071.stdout:397: writing smithi07183189-23 from 3307607 to 3462242 tid 3 2023-12-15T22:58:04.488 INFO:tasks.rados.rados.0.smithi071.stdout:398: delete oid 31 current snap is 44 2023-12-15T22:58:04.522 INFO:tasks.rados.rados.0.smithi071.stdout:394: finishing write tid 1 to smithi07183189-4 2023-12-15T22:58:04.535 INFO:tasks.rados.rados.0.smithi071.stdout:394: finishing write tid 2 to smithi07183189-4 2023-12-15T22:58:04.539 INFO:tasks.rados.rados.0.smithi071.stdout:397: finishing write tid 1 to smithi07183189-23 2023-12-15T22:58:04.540 INFO:tasks.rados.rados.0.smithi071.stdout:396: expect (ObjNum 105 snap 9 seq_num 105) 2023-12-15T22:58:04.775 INFO:tasks.rados.rados.0.smithi071.stdout:394: finishing write tid 3 to smithi07183189-4 2023-12-15T22:58:04.775 INFO:tasks.rados.rados.0.smithi071.stdout:397: finishing write tid 2 to smithi07183189-23 2023-12-15T22:58:04.775 INFO:tasks.rados.rados.0.smithi071.stdout:394: finishing write tid 4 to smithi07183189-4 2023-12-15T22:58:04.775 INFO:tasks.rados.rados.0.smithi071.stdout:394: finishing write tid 5 to smithi07183189-4 2023-12-15T22:58:04.775 INFO:tasks.rados.rados.0.smithi071.stdout:394: oid 4 updating version 0 to 133 2023-12-15T22:58:04.775 INFO:tasks.rados.rados.0.smithi071.stdout:394: oid 4 version 133 is already newer than 131 2023-12-15T22:58:04.775 INFO:tasks.rados.rados.0.smithi071.stdout:394: oid 4 version 133 is already newer than 132 2023-12-15T22:58:04.776 INFO:tasks.rados.rados.0.smithi071.stdout:394: oid 4 version 133 is already newer than 131 2023-12-15T22:58:04.776 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 4 v 133 (ObjNum 205 snap 44 seq_num 205) dirty exists 2023-12-15T22:58:04.776 INFO:tasks.rados.rados.0.smithi071.stdout:394: left oid 4 (ObjNum 205 snap 44 seq_num 205) 2023-12-15T22:58:04.776 INFO:tasks.rados.rados.0.smithi071.stdout:397: finishing write tid 3 to smithi07183189-23 2023-12-15T22:58:04.776 INFO:tasks.rados.rados.0.smithi071.stdout:394: done (4 left) 2023-12-15T22:58:04.776 INFO:tasks.rados.rados.0.smithi071.stdout:395: done (3 left) 2023-12-15T22:58:04.776 INFO:tasks.rados.rados.0.smithi071.stdout:396: done (2 left) 2023-12-15T22:58:04.776 INFO:tasks.rados.rados.0.smithi071.stdout:398: done (1 left) 2023-12-15T22:58:04.776 INFO:tasks.rados.rados.0.smithi071.stdout:399: write oid 50 current snap is 44 2023-12-15T22:58:04.776 INFO:tasks.rados.rados.0.smithi071.stdout:397: finishing write tid 4 to smithi07183189-23 2023-12-15T22:58:04.776 INFO:tasks.rados.rados.0.smithi071.stdout:397: finishing write tid 5 to smithi07183189-23 2023-12-15T22:58:04.776 INFO:tasks.rados.rados.0.smithi071.stdout:397: oid 23 updating version 0 to 59 2023-12-15T22:58:04.776 INFO:tasks.rados.rados.0.smithi071.stdout:397: oid 23 version 59 is already newer than 58 2023-12-15T22:58:04.776 INFO:tasks.rados.rados.0.smithi071.stdout:397: oid 23 version 59 is already newer than 57 2023-12-15T22:58:04.776 INFO:tasks.rados.rados.0.smithi071.stdout:397: oid 23 updating version 59 to 60 2023-12-15T22:58:04.777 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 23 v 60 (ObjNum 206 snap 44 seq_num 206) dirty exists 2023-12-15T22:58:04.777 INFO:tasks.rados.rados.0.smithi071.stdout:397: left oid 23 (ObjNum 206 snap 44 seq_num 206) 2023-12-15T22:58:04.777 INFO:tasks.rados.rados.0.smithi071.stdout:399: seq_num 208 ranges {695856=793323,2171784=534399,3362453=133403} 2023-12-15T22:58:04.782 INFO:tasks.rados.rados.0.smithi071.stdout:399: writing smithi07183189-50 from 695856 to 1489179 tid 1 2023-12-15T22:58:04.789 INFO:tasks.rados.rados.0.smithi071.stdout:399: writing smithi07183189-50 from 2171784 to 2706183 tid 2 2023-12-15T22:58:04.794 INFO:tasks.rados.rados.0.smithi071.stdout:399: writing smithi07183189-50 from 3362453 to 3495856 tid 3 2023-12-15T22:58:04.794 INFO:tasks.rados.rados.0.smithi071.stdout:397: done (1 left) 2023-12-15T22:58:04.794 INFO:tasks.rados.rados.0.smithi071.stdout:400: write oid 24 current snap is 44 2023-12-15T22:58:04.794 INFO:tasks.rados.rados.0.smithi071.stdout:400: seq_num 209 ranges {712663=468572,1713913=678057,2873523=591733} 2023-12-15T22:58:04.800 INFO:tasks.rados.rados.0.smithi071.stdout:400: writing smithi07183189-24 from 712663 to 1181235 tid 1 2023-12-15T22:58:04.807 INFO:tasks.rados.rados.0.smithi071.stdout:400: writing smithi07183189-24 from 1713913 to 2391970 tid 2 2023-12-15T22:58:04.813 INFO:tasks.rados.rados.0.smithi071.stdout:400: writing smithi07183189-24 from 2873523 to 3465256 tid 3 2023-12-15T22:58:04.813 INFO:tasks.rados.rados.0.smithi071.stdout:401: read oid 15 snap -1 2023-12-15T22:58:04.813 INFO:tasks.rados.rados.0.smithi071.stdout:401: expect (ObjNum 197 snap 40 seq_num 197) 2023-12-15T22:58:04.813 INFO:tasks.rados.rados.0.smithi071.stdout:402: snap_create 2023-12-15T22:58:04.815 INFO:tasks.rados.rados.0.smithi071.stdout:399: finishing write tid 1 to smithi07183189-50 2023-12-15T22:58:04.844 INFO:tasks.rados.rados.0.smithi071.stdout:400: finishing write tid 1 to smithi07183189-24 2023-12-15T22:58:04.847 INFO:tasks.rados.rados.0.smithi071.stdout:399: finishing write tid 2 to smithi07183189-50 2023-12-15T22:58:04.866 INFO:tasks.rados.rados.0.smithi071.stdout:401: expect (ObjNum 197 snap 40 seq_num 197) 2023-12-15T22:58:05.000 INFO:tasks.rados.rados.0.smithi071.stdout:399: finishing write tid 3 to smithi07183189-50 2023-12-15T22:58:05.000 INFO:tasks.rados.rados.0.smithi071.stdout:399: finishing write tid 4 to smithi07183189-50 2023-12-15T22:58:05.000 INFO:tasks.rados.rados.0.smithi071.stdout:399: finishing write tid 5 to smithi07183189-50 2023-12-15T22:58:05.000 INFO:tasks.rados.rados.0.smithi071.stdout:399: oid 50 updating version 0 to 195 2023-12-15T22:58:05.000 INFO:tasks.rados.rados.0.smithi071.stdout:399: oid 50 version 195 is already newer than 192 2023-12-15T22:58:05.000 INFO:tasks.rados.rados.0.smithi071.stdout:399: oid 50 version 195 is already newer than 193 2023-12-15T22:58:05.000 INFO:tasks.rados.rados.0.smithi071.stdout:399: oid 50 version 195 is already newer than 195 2023-12-15T22:58:05.001 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 195 (ObjNum 208 snap 44 seq_num 208) dirty exists 2023-12-15T22:58:05.001 INFO:tasks.rados.rados.0.smithi071.stdout:399: left oid 50 (ObjNum 208 snap 44 seq_num 208) 2023-12-15T22:58:05.001 INFO:tasks.rados.rados.0.smithi071.stdout:400: finishing write tid 2 to smithi07183189-24 2023-12-15T22:58:05.001 INFO:tasks.rados.rados.0.smithi071.stdout:400: finishing write tid 3 to smithi07183189-24 2023-12-15T22:58:05.001 INFO:tasks.rados.rados.0.smithi071.stdout:400: finishing write tid 4 to smithi07183189-24 2023-12-15T22:58:05.001 INFO:tasks.rados.rados.0.smithi071.stdout:400: finishing write tid 5 to smithi07183189-24 2023-12-15T22:58:05.001 INFO:tasks.rados.rados.0.smithi071.stdout:400: oid 24 updating version 0 to 22 2023-12-15T22:58:05.001 INFO:tasks.rados.rados.0.smithi071.stdout:400: oid 24 version 22 is already newer than 20 2023-12-15T22:58:05.001 INFO:tasks.rados.rados.0.smithi071.stdout:400: oid 24 version 22 is already newer than 22 2023-12-15T22:58:05.001 INFO:tasks.rados.rados.0.smithi071.stdout:400: oid 24 version 22 is already newer than 19 2023-12-15T22:58:05.001 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 22 (ObjNum 209 snap 44 seq_num 209) dirty exists 2023-12-15T22:58:05.001 INFO:tasks.rados.rados.0.smithi071.stdout:400: left oid 24 (ObjNum 209 snap 44 seq_num 209) 2023-12-15T22:58:05.288 INFO:tasks.rados.rados.0.smithi071.stdout:399: done (3 left) 2023-12-15T22:58:05.289 INFO:tasks.rados.rados.0.smithi071.stdout:400: done (2 left) 2023-12-15T22:58:05.289 INFO:tasks.rados.rados.0.smithi071.stdout:401: done (1 left) 2023-12-15T22:58:05.289 INFO:tasks.rados.rados.0.smithi071.stdout:402: done (0 left) 2023-12-15T22:58:05.289 INFO:tasks.rados.rados.0.smithi071.stdout:403: read oid 17 snap -1 2023-12-15T22:58:05.289 INFO:tasks.rados.rados.0.smithi071.stdout:403: expect (ObjNum 202 snap 43 seq_num 202) 2023-12-15T22:58:05.289 INFO:tasks.rados.rados.0.smithi071.stdout:404: read oid 35 snap 37 2023-12-15T22:58:05.289 INFO:tasks.rados.rados.0.smithi071.stdout:404: expect (ObjNum 188 snap 36 seq_num 188) 2023-12-15T22:58:05.289 INFO:tasks.rados.rados.0.smithi071.stdout:405: read oid 8 snap -1 2023-12-15T22:58:05.289 INFO:tasks.rados.rados.0.smithi071.stdout:405: expect deleted 2023-12-15T22:58:05.289 INFO:tasks.rados.rados.0.smithi071.stdout:406: snap_remove snap 44 2023-12-15T22:58:05.531 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:58:05.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:05.634+0000 7f6d75d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:05.638 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:05.638+0000 7f6d75d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:06.292 INFO:tasks.rados.rados.0.smithi071.stdout:406: done (3 left) 2023-12-15T22:58:06.292 INFO:tasks.rados.rados.0.smithi071.stdout:407: delete oid 4 current snap is 45 2023-12-15T22:58:06.292 INFO:tasks.rados.rados.0.smithi071.stdout:404: expect (ObjNum 188 snap 36 seq_num 188) 2023-12-15T22:58:06.535 INFO:tasks.rados.rados.0.smithi071.stdout:403: expect (ObjNum 202 snap 43 seq_num 202) 2023-12-15T22:58:06.755 INFO:tasks.rados.rados.0.smithi071.stdout:403: done (3 left) 2023-12-15T22:58:06.755 INFO:tasks.rados.rados.0.smithi071.stdout:404: done (2 left) 2023-12-15T22:58:06.755 INFO:tasks.rados.rados.0.smithi071.stdout:405: done (1 left) 2023-12-15T22:58:06.755 INFO:tasks.rados.rados.0.smithi071.stdout:407: done (0 left) 2023-12-15T22:58:06.755 INFO:tasks.rados.rados.0.smithi071.stdout:408: write oid 13 current snap is 45 2023-12-15T22:58:06.756 INFO:tasks.rados.rados.0.smithi071.stdout:408: seq_num 211 ranges {746277=619070,1830877=681726,2928369=617908} 2023-12-15T22:58:06.762 INFO:tasks.rados.rados.0.smithi071.stdout:408: writing smithi07183189-13 from 746277 to 1365347 tid 1 2023-12-15T22:58:06.769 INFO:tasks.rados.rados.0.smithi071.stdout:408: writing smithi07183189-13 from 1830877 to 2512603 tid 2 2023-12-15T22:58:06.774 INFO:tasks.rados.rados.0.smithi071.stdout:408: writing smithi07183189-13 from 2928369 to 3546277 tid 3 2023-12-15T22:58:06.774 INFO:tasks.rados.rados.0.smithi071.stdout:409: write (excl) oid 14 current snap is 45 2023-12-15T22:58:06.775 INFO:tasks.rados.rados.0.smithi071.stdout:409: seq_num 212 ranges {763084=694319,1889359=541737,2955792=541902} 2023-12-15T22:58:06.782 INFO:tasks.rados.rados.0.smithi071.stdout:409: writing smithi07183189-14 from 763084 to 1457403 tid 1 2023-12-15T22:58:06.787 INFO:tasks.rados.rados.0.smithi071.stdout:409: writing smithi07183189-14 from 1889359 to 2431096 tid 2 2023-12-15T22:58:06.793 INFO:tasks.rados.rados.0.smithi071.stdout:409: writing smithi07183189-14 from 2955792 to 3497694 tid 3 2023-12-15T22:58:06.794 INFO:tasks.rados.rados.0.smithi071.stdout:410: read oid 27 snap -1 2023-12-15T22:58:06.794 INFO:tasks.rados.rados.0.smithi071.stdout:410: expect deleted 2023-12-15T22:58:06.794 INFO:tasks.rados.rados.0.smithi071.stdout:411: snap_remove snap 37 2023-12-15T22:58:07.312 INFO:tasks.rados.rados.0.smithi071.stdout:411: done (3 left) 2023-12-15T22:58:07.312 INFO:tasks.rados.rados.0.smithi071.stdout:412: write (excl) oid 31 current snap is 45 2023-12-15T22:58:07.312 INFO:tasks.rados.rados.0.smithi071.stdout:412: seq_num 213 ranges {779891=485921,1947841=685395,3383215=196676} 2023-12-15T22:58:07.320 INFO:tasks.rados.rados.0.smithi071.stdout:412: writing smithi07183189-31 from 779891 to 1265812 tid 1 2023-12-15T22:58:07.329 INFO:tasks.rados.rados.0.smithi071.stdout:412: writing smithi07183189-31 from 1947841 to 2633236 tid 2 2023-12-15T22:58:07.335 INFO:tasks.rados.rados.0.smithi071.stdout:412: writing smithi07183189-31 from 3383215 to 3579891 tid 3 2023-12-15T22:58:07.335 INFO:tasks.rados.rados.0.smithi071.stdout:413: snap_create 2023-12-15T22:58:07.335 INFO:tasks.rados.rados.0.smithi071.stdout:408: finishing write tid 1 to smithi07183189-13 2023-12-15T22:58:07.335 INFO:tasks.rados.rados.0.smithi071.stdout:408: finishing write tid 2 to smithi07183189-13 2023-12-15T22:58:07.335 INFO:tasks.rados.rados.0.smithi071.stdout:408: finishing write tid 3 to smithi07183189-13 2023-12-15T22:58:07.335 INFO:tasks.rados.rados.0.smithi071.stdout:408: finishing write tid 4 to smithi07183189-13 2023-12-15T22:58:07.335 INFO:tasks.rados.rados.0.smithi071.stdout:408: finishing write tid 5 to smithi07183189-13 2023-12-15T22:58:07.335 INFO:tasks.rados.rados.0.smithi071.stdout:408: oid 13 updating version 0 to 70 2023-12-15T22:58:07.336 INFO:tasks.rados.rados.0.smithi071.stdout:408: oid 13 version 70 is already newer than 70 2023-12-15T22:58:07.336 INFO:tasks.rados.rados.0.smithi071.stdout:408: oid 13 updating version 70 to 71 2023-12-15T22:58:07.336 INFO:tasks.rados.rados.0.smithi071.stdout:408: oid 13 updating version 71 to 72 2023-12-15T22:58:07.336 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 72 (ObjNum 211 snap 45 seq_num 211) dirty exists 2023-12-15T22:58:07.336 INFO:tasks.rados.rados.0.smithi071.stdout:408: left oid 13 (ObjNum 211 snap 45 seq_num 211) 2023-12-15T22:58:07.336 INFO:tasks.rados.rados.0.smithi071.stdout:409: finishing write tid 1 to smithi07183189-14 2023-12-15T22:58:07.336 INFO:tasks.rados.rados.0.smithi071.stdout:409: finishing write tid 2 to smithi07183189-14 2023-12-15T22:58:07.336 INFO:tasks.rados.rados.0.smithi071.stdout:409: finishing write tid 3 to smithi07183189-14 2023-12-15T22:58:07.336 INFO:tasks.rados.rados.0.smithi071.stdout:409: finishing write tid 4 to smithi07183189-14 2023-12-15T22:58:07.336 INFO:tasks.rados.rados.0.smithi071.stdout:409: finishing write tid 5 to smithi07183189-14 2023-12-15T22:58:07.336 INFO:tasks.rados.rados.0.smithi071.stdout:409: oid 14 updating version 0 to 74 2023-12-15T22:58:07.336 INFO:tasks.rados.rados.0.smithi071.stdout:409: oid 14 updating version 74 to 76 2023-12-15T22:58:07.336 INFO:tasks.rados.rados.0.smithi071.stdout:409: oid 14 version 76 is already newer than 75 2023-12-15T22:58:07.336 INFO:tasks.rados.rados.0.smithi071.stdout:409: oid 14 updating version 76 to 77 2023-12-15T22:58:07.336 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 77 (ObjNum 212 snap 45 seq_num 212) dirty exists 2023-12-15T22:58:07.336 INFO:tasks.rados.rados.0.smithi071.stdout:409: left oid 14 (ObjNum 212 snap 45 seq_num 212) 2023-12-15T22:58:07.396 INFO:tasks.rados.rados.0.smithi071.stdout:412: finishing write tid 1 to smithi07183189-31 2023-12-15T22:58:07.414 INFO:tasks.rados.rados.0.smithi071.stdout:412: finishing write tid 2 to smithi07183189-31 2023-12-15T22:58:07.424 INFO:tasks.rados.rados.0.smithi071.stdout:412: finishing write tid 3 to smithi07183189-31 2023-12-15T22:58:07.432 INFO:tasks.rados.rados.0.smithi071.stdout:412: finishing write tid 4 to smithi07183189-31 2023-12-15T22:58:07.433 INFO:tasks.rados.rados.0.smithi071.stdout:412: finishing write tid 5 to smithi07183189-31 2023-12-15T22:58:07.433 INFO:tasks.rados.rados.0.smithi071.stdout:412: oid 31 updating version 0 to 65 2023-12-15T22:58:07.433 INFO:tasks.rados.rados.0.smithi071.stdout:412: oid 31 updating version 65 to 67 2023-12-15T22:58:07.433 INFO:tasks.rados.rados.0.smithi071.stdout:412: oid 31 version 67 is already newer than 66 2023-12-15T22:58:07.433 INFO:tasks.rados.rados.0.smithi071.stdout:412: oid 31 updating version 67 to 68 2023-12-15T22:58:07.433 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 68 (ObjNum 213 snap 45 seq_num 213) dirty exists 2023-12-15T22:58:07.433 INFO:tasks.rados.rados.0.smithi071.stdout:412: left oid 31 (ObjNum 213 snap 45 seq_num 213) 2023-12-15T22:58:07.512 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T22:58:07.512 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T22:58:07.512 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T22:58:07.622 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:07.618+0000 7f3b5839e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:07.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:07.625+0000 7f3b5839e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:07.706 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:58:07.802 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:58:07.803 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-12-15T22:58:07.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:07.808+0000 7fe5b599e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:07.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:07.813+0000 7fe5b599e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:07.814 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":143,"stamp":"2023-12-15T22:58:07.307491+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":119583697,"num_objects":142,"num_object_clones":80,"num_object_copies":426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":939,"num_read_kb":587997,"num_write":845,"num_write_kb":206316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1066,"ondisk_log_size":1066,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":386016,"kb_used_data":270400,"kb_used_omap":0,"kb_used_meta":115584,"kb_avail":377101344,"statfs":{"total":386547056640,"available":386151776256,"internally_reserved":0,"allocated":276889600,"data_stored":294732273,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":118358016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1243718,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":77,"num_read_kb":84535,"num_write":18,"num_write_kb":5805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.064570"},"pg_stats":[{"pgid":"3.f","version":"73'24","reported_seq":208,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-15T22:58:05.304349+0000","last_change":"2023-12-15T22:58:03.305831+0000","last_active":"2023-12-15T22:58:05.304349+0000","last_peered":"2023-12-15T22:58:05.304349+0000","last_clean":"2023-12-15T22:58:05.304349+0000","last_became_active":"2023-12-15T22:56:39.503797+0000","last_became_peered":"2023-12-15T22:56:39.503797+0000","last_unstale":"2023-12-15T22:58:05.304349+0000","last_undegraded":"2023-12-15T22:58:05.304349+0000","last_fullsized":"2023-12-15T22:58:05.304349+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4941225,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":32,"num_read_kb":24675,"num_write":17,"num_write_kb":4759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":178,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-15T22:58:04.483741+0000","last_change":"2023-12-15T22:58:03.276501+0000","last_active":"2023-12-15T22:58:04.483741+0000","last_peered":"2023-12-15T22:58:04.483741+0000","last_clean":"2023-12-15T22:58:04.483741+0000","last_became_active":"2023-12-15T22:56:39.552856+0000","last_became_peered":"2023-12-15T22:56:39.552856+0000","last_unstale":"2023-12-15T22:58:04.483741+0000","last_undegraded":"2023-12-15T22:58:04.483741+0000","last_fullsized":"2023-12-15T22:58:04.483741+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"98'131","reported_seq":326,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-15T22:58:04.499635+0000","last_change":"2023-12-15T22:58:03.316063+0000","last_active":"2023-12-15T22:58:04.499635+0000","last_peered":"2023-12-15T22:58:04.499635+0000","last_clean":"2023-12-15T22:58:04.499635+0000","last_became_active":"2023-12-15T22:56:39.495490+0000","last_became_peered":"2023-12-15T22:56:39.495490+0000","last_unstale":"2023-12-15T22:58:04.499635+0000","last_undegraded":"2023-12-15T22:58:04.499635+0000","last_fullsized":"2023-12-15T22:58:04.499635+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18134753,"num_objects":20,"num_object_clones":11,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":123,"num_read_kb":91139,"num_write":107,"num_write_kb":30928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"93'88","reported_seq":272,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-15T22:58:05.332214+0000","last_change":"2023-12-15T22:58:03.307122+0000","last_active":"2023-12-15T22:58:05.332214+0000","last_peered":"2023-12-15T22:58:05.332214+0000","last_clean":"2023-12-15T22:58:05.332214+0000","last_became_active":"2023-12-15T22:56:39.579234+0000","last_became_peered":"2023-12-15T22:56:39.579234+0000","last_unstale":"2023-12-15T22:58:05.332214+0000","last_undegraded":"2023-12-15T22:58:05.332214+0000","last_fullsized":"2023-12-15T22:58:05.332214+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9361499,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":80,"num_read_kb":59758,"num_write":66,"num_write_kb":18365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"93'48","reported_seq":223,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-15T22:58:05.303367+0000","last_change":"2023-12-15T22:58:03.499839+0000","last_active":"2023-12-15T22:58:05.303367+0000","last_peered":"2023-12-15T22:58:05.303367+0000","last_clean":"2023-12-15T22:58:05.303367+0000","last_became_active":"2023-12-15T22:56:39.572176+0000","last_became_peered":"2023-12-15T22:56:39.572176+0000","last_unstale":"2023-12-15T22:58:05.303367+0000","last_undegraded":"2023-12-15T22:58:05.303367+0000","last_fullsized":"2023-12-15T22:58:05.303367+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9647218,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":8583,"num_write":36,"num_write_kb":10640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"90'54","reported_seq":220,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-15T22:58:05.298705+0000","last_change":"2023-12-15T22:58:03.307544+0000","last_active":"2023-12-15T22:58:05.298705+0000","last_peered":"2023-12-15T22:58:05.298705+0000","last_clean":"2023-12-15T22:58:05.298705+0000","last_became_active":"2023-12-15T22:56:39.516361+0000","last_became_peered":"2023-12-15T22:56:39.516361+0000","last_unstale":"2023-12-15T22:58:05.298705+0000","last_undegraded":"2023-12-15T22:58:05.298705+0000","last_fullsized":"2023-12-15T22:58:05.298705+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3109295,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":25,"num_read_kb":19608,"num_write":35,"num_write_kb":10277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"98'195","reported_seq":357,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-15T22:58:05.300121+0000","last_change":"2023-12-15T22:58:03.308652+0000","last_active":"2023-12-15T22:58:05.300121+0000","last_peered":"2023-12-15T22:58:05.300121+0000","last_clean":"2023-12-15T22:58:05.300121+0000","last_became_active":"2023-12-15T22:56:39.519748+0000","last_became_peered":"2023-12-15T22:56:39.519748+0000","last_unstale":"2023-12-15T22:58:05.300121+0000","last_undegraded":"2023-12-15T22:58:05.300121+0000","last_fullsized":"2023-12-15T22:58:05.300121+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16168334,"num_objects":29,"num_object_clones":19,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":148,"num_read_kb":91858,"num_write":124,"num_write_kb":35773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"82'46","reported_seq":238,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-15T22:58:04.469462+0000","last_change":"2023-12-15T22:58:03.303511+0000","last_active":"2023-12-15T22:58:04.469462+0000","last_peered":"2023-12-15T22:58:04.469462+0000","last_clean":"2023-12-15T22:58:04.469462+0000","last_became_active":"2023-12-15T22:56:39.543179+0000","last_became_peered":"2023-12-15T22:56:39.543179+0000","last_unstale":"2023-12-15T22:58:04.469462+0000","last_undegraded":"2023-12-15T22:58:04.469462+0000","last_fullsized":"2023-12-15T22:58:04.469462+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8857223,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":62,"num_read_kb":55425,"num_write":40,"num_write_kb":10468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":274,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-15T22:58:05.303665+0000","last_change":"2023-12-15T22:56:32.607017+0000","last_active":"2023-12-15T22:58:05.303665+0000","last_peered":"2023-12-15T22:58:05.303665+0000","last_clean":"2023-12-15T22:58:05.303665+0000","last_became_active":"2023-12-15T22:56:32.602384+0000","last_became_peered":"2023-12-15T22:56:32.602384+0000","last_unstale":"2023-12-15T22:58:05.303665+0000","last_undegraded":"2023-12-15T22:58:05.303665+0000","last_fullsized":"2023-12-15T22:58:05.303665+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_clean_scrub_stamp":"2023-12-15T22:56:31.525572+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":143,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-15T22:58:04.470507+0000","last_change":"2023-12-15T22:56:33.627481+0000","last_active":"2023-12-15T22:58:04.470507+0000","last_peered":"2023-12-15T22:58:04.470507+0000","last_clean":"2023-12-15T22:58:04.470507+0000","last_became_active":"2023-12-15T22:56:33.624689+0000","last_became_peered":"2023-12-15T22:56:33.624689+0000","last_unstale":"2023-12-15T22:58:04.470507+0000","last_undegraded":"2023-12-15T22:58:04.470507+0000","last_fullsized":"2023-12-15T22:58:04.470507+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"97'73","reported_seq":250,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-15T22:58:04.465282+0000","last_change":"2023-12-15T22:58:03.296440+0000","last_active":"2023-12-15T22:58:04.465282+0000","last_peered":"2023-12-15T22:58:04.465282+0000","last_clean":"2023-12-15T22:58:04.465282+0000","last_became_active":"2023-12-15T22:56:39.529359+0000","last_became_peered":"2023-12-15T22:56:39.529359+0000","last_unstale":"2023-12-15T22:58:04.465282+0000","last_undegraded":"2023-12-15T22:58:04.465282+0000","last_fullsized":"2023-12-15T22:58:04.465282+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8218623,"num_objects":9,"num_object_clones":4,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":43,"num_read_kb":16494,"num_write":61,"num_write_kb":17671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":147,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-15T22:58:05.309742+0000","last_change":"2023-12-15T22:56:33.655797+0000","last_active":"2023-12-15T22:58:05.309742+0000","last_peered":"2023-12-15T22:58:05.309742+0000","last_clean":"2023-12-15T22:58:05.309742+0000","last_became_active":"2023-12-15T22:56:33.652885+0000","last_became_peered":"2023-12-15T22:56:33.652885+0000","last_unstale":"2023-12-15T22:58:05.309742+0000","last_undegraded":"2023-12-15T22:58:05.309742+0000","last_fullsized":"2023-12-15T22:58:05.309742+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"95'67","reported_seq":258,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-15T22:58:05.302878+0000","last_change":"2023-12-15T22:58:03.502923+0000","last_active":"2023-12-15T22:58:05.302878+0000","last_peered":"2023-12-15T22:58:05.302878+0000","last_clean":"2023-12-15T22:58:05.302878+0000","last_became_active":"2023-12-15T22:56:39.543509+0000","last_became_peered":"2023-12-15T22:56:39.543509+0000","last_unstale":"2023-12-15T22:58:05.302878+0000","last_undegraded":"2023-12-15T22:58:05.302878+0000","last_fullsized":"2023-12-15T22:58:05.302878+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12353079,"num_objects":11,"num_object_clones":5,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":79,"num_read_kb":58887,"num_write":47,"num_write_kb":13212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":143,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-15T22:58:04.467205+0000","last_change":"2023-12-15T22:56:33.600485+0000","last_active":"2023-12-15T22:58:04.467205+0000","last_peered":"2023-12-15T22:58:04.467205+0000","last_clean":"2023-12-15T22:58:04.467205+0000","last_became_active":"2023-12-15T22:56:33.597320+0000","last_became_peered":"2023-12-15T22:56:33.597320+0000","last_unstale":"2023-12-15T22:58:04.467205+0000","last_undegraded":"2023-12-15T22:58:04.467205+0000","last_fullsized":"2023-12-15T22:58:04.467205+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"55'26","reported_seq":206,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-15T22:58:05.302058+0000","last_change":"2023-12-15T22:58:03.500070+0000","last_active":"2023-12-15T22:58:05.302058+0000","last_peered":"2023-12-15T22:58:05.302058+0000","last_clean":"2023-12-15T22:58:05.302058+0000","last_became_active":"2023-12-15T22:56:39.560980+0000","last_became_peered":"2023-12-15T22:56:39.560980+0000","last_unstale":"2023-12-15T22:58:05.302058+0000","last_undegraded":"2023-12-15T22:58:05.302058+0000","last_fullsized":"2023-12-15T22:58:05.302058+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5747961,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":13740,"num_write":21,"num_write_kb":6063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":147,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-15T22:58:05.308723+0000","last_change":"2023-12-15T22:56:33.648541+0000","last_active":"2023-12-15T22:58:05.308723+0000","last_peered":"2023-12-15T22:58:05.308723+0000","last_clean":"2023-12-15T22:58:05.308723+0000","last_became_active":"2023-12-15T22:56:33.643681+0000","last_became_peered":"2023-12-15T22:56:33.643681+0000","last_unstale":"2023-12-15T22:58:05.308723+0000","last_undegraded":"2023-12-15T22:58:05.308723+0000","last_fullsized":"2023-12-15T22:58:05.308723+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"93'77","reported_seq":255,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-15T22:58:04.498575+0000","last_change":"2023-12-15T22:58:03.272318+0000","last_active":"2023-12-15T22:58:04.498575+0000","last_peered":"2023-12-15T22:58:04.498575+0000","last_clean":"2023-12-15T22:58:04.498575+0000","last_became_active":"2023-12-15T22:56:39.506510+0000","last_became_peered":"2023-12-15T22:56:39.506510+0000","last_unstale":"2023-12-15T22:58:04.498575+0000","last_undegraded":"2023-12-15T22:58:04.498575+0000","last_fullsized":"2023-12-15T22:58:04.498575+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5680766,"num_objects":12,"num_object_clones":7,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":68,"num_read_kb":48987,"num_write":49,"num_write_kb":14685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"90'17","reported_seq":189,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-15T22:58:04.464587+0000","last_change":"2023-12-15T22:58:03.311139+0000","last_active":"2023-12-15T22:58:04.464587+0000","last_peered":"2023-12-15T22:58:04.464587+0000","last_clean":"2023-12-15T22:58:04.464587+0000","last_became_active":"2023-12-15T22:56:39.511159+0000","last_became_peered":"2023-12-15T22:56:39.511159+0000","last_unstale":"2023-12-15T22:58:04.464587+0000","last_undegraded":"2023-12-15T22:58:04.464587+0000","last_fullsized":"2023-12-15T22:58:04.464587+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3210137,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":7,"num_read_kb":3,"num_write":14,"num_write_kb":3818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":145,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-15T22:58:04.493254+0000","last_change":"2023-12-15T22:56:33.633890+0000","last_active":"2023-12-15T22:58:04.493254+0000","last_peered":"2023-12-15T22:58:04.493254+0000","last_clean":"2023-12-15T22:58:04.493254+0000","last_became_active":"2023-12-15T22:56:33.631134+0000","last_became_peered":"2023-12-15T22:56:33.631134+0000","last_unstale":"2023-12-15T22:58:04.493254+0000","last_undegraded":"2023-12-15T22:58:04.493254+0000","last_fullsized":"2023-12-15T22:58:04.493254+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"54'47","reported_seq":221,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-15T22:58:04.526410+0000","last_change":"2023-12-15T22:58:03.274244+0000","last_active":"2023-12-15T22:58:04.526410+0000","last_peered":"2023-12-15T22:58:04.526410+0000","last_clean":"2023-12-15T22:58:04.526410+0000","last_became_active":"2023-12-15T22:56:39.532730+0000","last_became_peered":"2023-12-15T22:56:39.532730+0000","last_unstale":"2023-12-15T22:58:04.526410+0000","last_undegraded":"2023-12-15T22:58:04.526410+0000","last_fullsized":"2023-12-15T22:58:04.526410+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3764735,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":41,"num_read_kb":38478,"num_write":27,"num_write_kb":8830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":145,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-15T22:58:05.299376+0000","last_change":"2023-12-15T22:56:33.655736+0000","last_active":"2023-12-15T22:58:05.299376+0000","last_peered":"2023-12-15T22:58:05.299376+0000","last_clean":"2023-12-15T22:58:05.299376+0000","last_became_active":"2023-12-15T22:56:33.652917+0000","last_became_peered":"2023-12-15T22:56:33.652917+0000","last_unstale":"2023-12-15T22:58:05.299376+0000","last_undegraded":"2023-12-15T22:58:05.299376+0000","last_fullsized":"2023-12-15T22:58:05.299376+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"98'62","reported_seq":255,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-15T22:58:05.301404+0000","last_change":"2023-12-15T22:58:03.499861+0000","last_active":"2023-12-15T22:58:05.301404+0000","last_peered":"2023-12-15T22:58:05.301404+0000","last_clean":"2023-12-15T22:58:05.301404+0000","last_became_active":"2023-12-15T22:56:39.557252+0000","last_became_peered":"2023-12-15T22:56:39.557252+0000","last_unstale":"2023-12-15T22:58:05.301404+0000","last_undegraded":"2023-12-15T22:58:05.301404+0000","last_fullsized":"2023-12-15T22:58:05.301404+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6504309,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":61,"num_read_kb":46207,"num_write":51,"num_write_kb":15005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":145,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-15T22:58:04.501923+0000","last_change":"2023-12-15T22:56:33.632348+0000","last_active":"2023-12-15T22:58:04.501923+0000","last_peered":"2023-12-15T22:58:04.501923+0000","last_clean":"2023-12-15T22:58:04.501923+0000","last_became_active":"2023-12-15T22:56:33.629022+0000","last_became_peered":"2023-12-15T22:56:33.629022+0000","last_unstale":"2023-12-15T22:58:04.501923+0000","last_undegraded":"2023-12-15T22:58:04.501923+0000","last_fullsized":"2023-12-15T22:58:04.501923+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"93'14","reported_seq":197,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-15T22:58:04.484466+0000","last_change":"2023-12-15T22:58:03.276851+0000","last_active":"2023-12-15T22:58:04.484466+0000","last_peered":"2023-12-15T22:58:04.484466+0000","last_clean":"2023-12-15T22:58:04.484466+0000","last_became_active":"2023-12-15T22:56:39.535145+0000","last_became_peered":"2023-12-15T22:56:39.535145+0000","last_unstale":"2023-12-15T22:58:04.484466+0000","last_undegraded":"2023-12-15T22:58:04.484466+0000","last_fullsized":"2023-12-15T22:58:04.484466+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3294172,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":14064,"num_write":13,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":143,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-15T22:58:04.466640+0000","last_change":"2023-12-15T22:56:33.627005+0000","last_active":"2023-12-15T22:58:04.466640+0000","last_peered":"2023-12-15T22:58:04.466640+0000","last_clean":"2023-12-15T22:58:04.466640+0000","last_became_active":"2023-12-15T22:56:33.622181+0000","last_became_peered":"2023-12-15T22:56:33.622181+0000","last_unstale":"2023-12-15T22:58:04.466640+0000","last_undegraded":"2023-12-15T22:58:04.466640+0000","last_fullsized":"2023-12-15T22:58:04.466640+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":118993329,"num_objects":140,"num_object_clones":80,"num_object_copies":420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":836,"num_read_kb":587906,"num_write":708,"num_write_kb":204422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":969,"ondisk_log_size":969,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705759,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":106452,"kb_used_data":75468,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94265388,"statfs":{"total":96636764160,"available":96527757312,"internally_reserved":0,"allocated":77279232,"data_stored":82446199,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":83084,"kb_used_data":56260,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94288756,"statfs":{"total":96636764160,"available":96551686144,"internally_reserved":0,"allocated":57610240,"data_stored":61227826,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705758,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":88120,"kb_used_data":61296,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94283720,"statfs":{"total":96636764160,"available":96546529280,"internally_reserved":0,"allocated":62767104,"data_stored":66696474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705759,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":108360,"kb_used_data":77376,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94263480,"statfs":{"total":96636764160,"available":96525803520,"internally_reserved":0,"allocated":79233024,"data_stored":84361774,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-15T22:58:08.308 INFO:tasks.rados.rados.0.smithi071.stdout:408: done (4 left) 2023-12-15T22:58:08.308 INFO:tasks.rados.rados.0.smithi071.stdout:409: done (3 left) 2023-12-15T22:58:08.308 INFO:tasks.rados.rados.0.smithi071.stdout:410: done (2 left) 2023-12-15T22:58:08.308 INFO:tasks.rados.rados.0.smithi071.stdout:412: done (1 left) 2023-12-15T22:58:08.309 INFO:tasks.rados.rados.0.smithi071.stdout:413: done (0 left) 2023-12-15T22:58:08.309 INFO:tasks.rados.rados.0.smithi071.stdout:414: snap_remove snap 43 2023-12-15T22:58:09.310 INFO:tasks.rados.rados.0.smithi071.stdout:414: done (0 left) 2023-12-15T22:58:09.310 INFO:tasks.rados.rados.0.smithi071.stdout:415: read oid 22 snap -1 2023-12-15T22:58:09.310 INFO:tasks.rados.rados.0.smithi071.stdout:415: expect (ObjNum 195 snap 40 seq_num 195) 2023-12-15T22:58:09.310 INFO:tasks.rados.rados.0.smithi071.stdout:416: snap_remove snap 38 2023-12-15T22:58:09.880 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:58:09.987 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:09.986+0000 7f1054060640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:09.990 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:09.990+0000 7f1054060640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:10.310 INFO:tasks.rados.rados.0.smithi071.stdout:416: done (1 left) 2023-12-15T22:58:10.310 INFO:tasks.rados.rados.0.smithi071.stdout:417: write (excl) oid 14 current snap is 46 2023-12-15T22:58:10.310 INFO:tasks.rados.rados.0.smithi071.stdout:417: seq_num 214 ranges {796698=561170,2006323=545406,3126991=469707} 2023-12-15T22:58:10.319 INFO:tasks.rados.rados.0.smithi071.stdout:417: writing smithi07183189-14 from 796698 to 1357868 tid 1 2023-12-15T22:58:10.327 INFO:tasks.rados.rados.0.smithi071.stdout:417: writing smithi07183189-14 from 2006323 to 2551729 tid 2 2023-12-15T22:58:10.335 INFO:tasks.rados.rados.0.smithi071.stdout:417: writing smithi07183189-14 from 3126991 to 3596698 tid 3 2023-12-15T22:58:10.335 INFO:tasks.rados.rados.0.smithi071.stdout:418: write oid 11 current snap is 46 2023-12-15T22:58:10.335 INFO:tasks.rados.rados.0.smithi071.stdout:418: seq_num 215 ranges {413505=636419,1664805=689064,3038061=492071} 2023-12-15T22:58:10.343 INFO:tasks.rados.rados.0.smithi071.stdout:418: writing smithi07183189-11 from 413505 to 1049924 tid 1 2023-12-15T22:58:10.351 INFO:tasks.rados.rados.0.smithi071.stdout:418: writing smithi07183189-11 from 1664805 to 2353869 tid 2 2023-12-15T22:58:10.359 INFO:tasks.rados.rados.0.smithi071.stdout:418: writing smithi07183189-11 from 3038061 to 3530132 tid 3 2023-12-15T22:58:10.359 INFO:tasks.rados.rados.0.smithi071.stdout:419: read oid 30 snap -1 2023-12-15T22:58:10.359 INFO:tasks.rados.rados.0.smithi071.stdout:419: expect (ObjNum 150 snap 20 seq_num 150) 2023-12-15T22:58:10.359 INFO:tasks.rados.rados.0.smithi071.stdout:420: snap_create 2023-12-15T22:58:10.359 INFO:tasks.rados.rados.0.smithi071.stdout:415: expect (ObjNum 195 snap 40 seq_num 195) 2023-12-15T22:58:10.494 INFO:tasks.rados.rados.0.smithi071.stdout:419: expect (ObjNum 150 snap 20 seq_num 150) 2023-12-15T22:58:10.595 INFO:tasks.rados.rados.0.smithi071.stdout:418: finishing write tid 1 to smithi07183189-11 2023-12-15T22:58:10.596 INFO:tasks.rados.rados.0.smithi071.stdout:417: finishing write tid 1 to smithi07183189-14 2023-12-15T22:58:10.596 INFO:tasks.rados.rados.0.smithi071.stdout:418: finishing write tid 2 to smithi07183189-11 2023-12-15T22:58:10.596 INFO:tasks.rados.rados.0.smithi071.stdout:418: finishing write tid 3 to smithi07183189-11 2023-12-15T22:58:10.596 INFO:tasks.rados.rados.0.smithi071.stdout:418: finishing write tid 4 to smithi07183189-11 2023-12-15T22:58:10.596 INFO:tasks.rados.rados.0.smithi071.stdout:417: finishing write tid 2 to smithi07183189-14 2023-12-15T22:58:10.596 INFO:tasks.rados.rados.0.smithi071.stdout:418: finishing write tid 5 to smithi07183189-11 2023-12-15T22:58:10.596 INFO:tasks.rados.rados.0.smithi071.stdout:418: oid 11 updating version 0 to 76 2023-12-15T22:58:10.596 INFO:tasks.rados.rados.0.smithi071.stdout:418: oid 11 updating version 76 to 78 2023-12-15T22:58:10.596 INFO:tasks.rados.rados.0.smithi071.stdout:418: oid 11 updating version 78 to 79 2023-12-15T22:58:10.596 INFO:tasks.rados.rados.0.smithi071.stdout:418: oid 11 version 79 is already newer than 78 2023-12-15T22:58:10.596 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 11 v 79 (ObjNum 215 snap 46 seq_num 215) dirty exists 2023-12-15T22:58:10.596 INFO:tasks.rados.rados.0.smithi071.stdout:418: left oid 11 (ObjNum 215 snap 46 seq_num 215) 2023-12-15T22:58:10.596 INFO:tasks.rados.rados.0.smithi071.stdout:417: finishing write tid 3 to smithi07183189-14 2023-12-15T22:58:10.596 INFO:tasks.rados.rados.0.smithi071.stdout:417: finishing write tid 4 to smithi07183189-14 2023-12-15T22:58:10.596 INFO:tasks.rados.rados.0.smithi071.stdout:417: finishing write tid 5 to smithi07183189-14 2023-12-15T22:58:10.597 INFO:tasks.rados.rados.0.smithi071.stdout:417: oid 14 updating version 0 to 87 2023-12-15T22:58:10.597 INFO:tasks.rados.rados.0.smithi071.stdout:417: oid 14 updating version 87 to 94 2023-12-15T22:58:10.597 INFO:tasks.rados.rados.0.smithi071.stdout:417: oid 14 version 94 is already newer than 92 2023-12-15T22:58:10.597 INFO:tasks.rados.rados.0.smithi071.stdout:417: oid 14 version 94 is already newer than 94 2023-12-15T22:58:10.597 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 94 (ObjNum 214 snap 46 seq_num 214) dirty exists 2023-12-15T22:58:10.597 INFO:tasks.rados.rados.0.smithi071.stdout:417: left oid 14 (ObjNum 214 snap 46 seq_num 214) 2023-12-15T22:58:11.321 INFO:tasks.rados.rados.0.smithi071.stdout:415: done (4 left) 2023-12-15T22:58:11.321 INFO:tasks.rados.rados.0.smithi071.stdout:417: done (3 left) 2023-12-15T22:58:11.321 INFO:tasks.rados.rados.0.smithi071.stdout:418: done (2 left) 2023-12-15T22:58:11.321 INFO:tasks.rados.rados.0.smithi071.stdout:419: done (1 left) 2023-12-15T22:58:11.321 INFO:tasks.rados.rados.0.smithi071.stdout:420: done (0 left) 2023-12-15T22:58:11.321 INFO:tasks.rados.rados.0.smithi071.stdout:421: write (excl) oid 13 current snap is 47 2023-12-15T22:58:11.322 INFO:tasks.rados.rados.0.smithi071.stdout:421: seq_num 216 ranges {430312=711668,1723287=549075,2781837=703343} 2023-12-15T22:58:11.327 INFO:tasks.rados.rados.0.smithi071.stdout:421: writing smithi07183189-13 from 430312 to 1141980 tid 1 2023-12-15T22:58:11.333 INFO:tasks.rados.rados.0.smithi071.stdout:421: writing smithi07183189-13 from 1723287 to 2272362 tid 2 2023-12-15T22:58:11.340 INFO:tasks.rados.rados.0.smithi071.stdout:421: writing smithi07183189-13 from 2781837 to 3485180 tid 3 2023-12-15T22:58:11.340 INFO:tasks.rados.rados.0.smithi071.stdout:422: write oid 48 current snap is 47 2023-12-15T22:58:11.340 INFO:tasks.rados.rados.0.smithi071.stdout:422: seq_num 217 ranges {447119=786917,1665416=692733,2976554=514615} 2023-12-15T22:58:11.346 INFO:tasks.rados.rados.0.smithi071.stdout:422: writing smithi07183189-48 from 447119 to 1234036 tid 1 2023-12-15T22:58:11.353 INFO:tasks.rados.rados.0.smithi071.stdout:422: writing smithi07183189-48 from 1665416 to 2358149 tid 2 2023-12-15T22:58:11.361 INFO:tasks.rados.rados.0.smithi071.stdout:422: writing smithi07183189-48 from 2976554 to 3491169 tid 3 2023-12-15T22:58:11.361 INFO:tasks.rados.rados.0.smithi071.stdout:423: write (excl) oid 12 current snap is 47 2023-12-15T22:58:11.361 INFO:tasks.rados.rados.0.smithi071.stdout:423: seq_num 218 ranges {463926=462166,1723898=436391,2603977=725887} 2023-12-15T22:58:11.367 INFO:tasks.rados.rados.0.smithi071.stdout:423: writing smithi07183189-12 from 463926 to 926092 tid 1 2023-12-15T22:58:11.375 INFO:tasks.rados.rados.0.smithi071.stdout:423: writing smithi07183189-12 from 1723898 to 2160289 tid 2 2023-12-15T22:58:11.383 INFO:tasks.rados.rados.0.smithi071.stdout:423: writing smithi07183189-12 from 2603977 to 3329864 tid 3 2023-12-15T22:58:11.383 INFO:tasks.rados.rados.0.smithi071.stdout:424: snap_remove snap 45 2023-12-15T22:58:12.059 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:58:12.169 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:12.168+0000 7ff279806640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:12.172 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:12.173+0000 7ff279806640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:12.243 INFO:tasks.rados.rados.0.smithi071.stdout:424: done (3 left) 2023-12-15T22:58:12.243 INFO:tasks.rados.rados.0.smithi071.stdout:425: snap_create 2023-12-15T22:58:12.243 INFO:tasks.rados.rados.0.smithi071.stdout:423: finishing write tid 1 to smithi07183189-12 2023-12-15T22:58:12.243 INFO:tasks.rados.rados.0.smithi071.stdout:422: finishing write tid 1 to smithi07183189-48 2023-12-15T22:58:12.243 INFO:tasks.rados.rados.0.smithi071.stdout:421: finishing write tid 1 to smithi07183189-13 2023-12-15T22:58:12.244 INFO:tasks.rados.rados.0.smithi071.stdout:423: finishing write tid 2 to smithi07183189-12 2023-12-15T22:58:12.244 INFO:tasks.rados.rados.0.smithi071.stdout:422: finishing write tid 2 to smithi07183189-48 2023-12-15T22:58:12.244 INFO:tasks.rados.rados.0.smithi071.stdout:421: finishing write tid 2 to smithi07183189-13 2023-12-15T22:58:12.244 INFO:tasks.rados.rados.0.smithi071.stdout:422: finishing write tid 3 to smithi07183189-48 2023-12-15T22:58:12.244 INFO:tasks.rados.rados.0.smithi071.stdout:423: finishing write tid 3 to smithi07183189-12 2023-12-15T22:58:12.244 INFO:tasks.rados.rados.0.smithi071.stdout:423: finishing write tid 4 to smithi07183189-12 2023-12-15T22:58:12.244 INFO:tasks.rados.rados.0.smithi071.stdout:423: finishing write tid 5 to smithi07183189-12 2023-12-15T22:58:12.244 INFO:tasks.rados.rados.0.smithi071.stdout:423: oid 12 updating version 0 to 52 2023-12-15T22:58:12.244 INFO:tasks.rados.rados.0.smithi071.stdout:423: oid 12 updating version 52 to 53 2023-12-15T22:58:12.244 INFO:tasks.rados.rados.0.smithi071.stdout:423: oid 12 updating version 53 to 55 2023-12-15T22:58:12.244 INFO:tasks.rados.rados.0.smithi071.stdout:423: oid 12 version 55 is already newer than 55 2023-12-15T22:58:12.244 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 12 v 55 (ObjNum 218 snap 47 seq_num 218) dirty exists 2023-12-15T22:58:12.244 INFO:tasks.rados.rados.0.smithi071.stdout:423: left oid 12 (ObjNum 218 snap 47 seq_num 218) 2023-12-15T22:58:12.244 INFO:tasks.rados.rados.0.smithi071.stdout:422: finishing write tid 4 to smithi07183189-48 2023-12-15T22:58:12.245 INFO:tasks.rados.rados.0.smithi071.stdout:422: finishing write tid 5 to smithi07183189-48 2023-12-15T22:58:12.245 INFO:tasks.rados.rados.0.smithi071.stdout:422: oid 48 updating version 0 to 83 2023-12-15T22:58:12.245 INFO:tasks.rados.rados.0.smithi071.stdout:422: oid 48 version 83 is already newer than 81 2023-12-15T22:58:12.245 INFO:tasks.rados.rados.0.smithi071.stdout:422: oid 48 version 83 is already newer than 83 2023-12-15T22:58:12.245 INFO:tasks.rados.rados.0.smithi071.stdout:422: oid 48 updating version 83 to 84 2023-12-15T22:58:12.245 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 84 (ObjNum 217 snap 47 seq_num 217) dirty exists 2023-12-15T22:58:12.245 INFO:tasks.rados.rados.0.smithi071.stdout:422: left oid 48 (ObjNum 217 snap 47 seq_num 217) 2023-12-15T22:58:12.245 INFO:tasks.rados.rados.0.smithi071.stdout:421: finishing write tid 3 to smithi07183189-13 2023-12-15T22:58:12.245 INFO:tasks.rados.rados.0.smithi071.stdout:421: finishing write tid 4 to smithi07183189-13 2023-12-15T22:58:12.245 INFO:tasks.rados.rados.0.smithi071.stdout:421: finishing write tid 5 to smithi07183189-13 2023-12-15T22:58:12.245 INFO:tasks.rados.rados.0.smithi071.stdout:421: oid 13 updating version 0 to 99 2023-12-15T22:58:12.245 INFO:tasks.rados.rados.0.smithi071.stdout:421: oid 13 version 99 is already newer than 97 2023-12-15T22:58:12.245 INFO:tasks.rados.rados.0.smithi071.stdout:421: oid 13 version 99 is already newer than 99 2023-12-15T22:58:12.245 INFO:tasks.rados.rados.0.smithi071.stdout:421: oid 13 version 99 is already newer than 96 2023-12-15T22:58:12.245 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 99 (ObjNum 216 snap 47 seq_num 216) dirty exists 2023-12-15T22:58:12.245 INFO:tasks.rados.rados.0.smithi071.stdout:421: left oid 13 (ObjNum 216 snap 47 seq_num 216) 2023-12-15T22:58:12.817 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T22:58:12.817 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T22:58:12.817 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-12-15T22:58:12.818 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-12-15T22:58:12.927 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:12.922+0000 7ff8cacad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:12.930 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:12.930+0000 7ff8cacad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:12.942 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:58:12.942 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:58:12.942 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:58:12.942 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:58:12.942 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:58:12.942 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:58:12.942 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:58:12.942 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:58:12.942 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:58:12.943 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:58:12.943 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:58:12.943 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:58:12.944 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:58:12.945 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:58:12.945 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-12-15T22:58:13.051 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:13.047+0000 7ff313ef0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:13.055 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:13.055+0000 7ff313ef0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:13.066 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:58:13.066 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:58:13.067 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:58:13.067 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:58:13.067 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:58:13.067 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:58:13.067 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:58:13.067 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:58:13.067 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:58:13.067 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:58:13.067 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:58:13.067 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:58:13.069 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:58:13.070 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:58:13.070 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-12-15T22:58:13.175 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:13.171+0000 7fc35ff5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:13.178 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:13.178+0000 7fc35ff5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:13.191 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:58:13.191 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:58:13.191 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:58:13.191 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:58:13.191 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:58:13.191 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:58:13.191 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:58:13.191 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:58:13.191 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:58:13.191 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:58:13.191 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:58:13.192 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:58:13.194 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:58:13.194 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:58:13.194 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-12-15T22:58:13.246 INFO:tasks.rados.rados.0.smithi071.stdout:421: done (3 left) 2023-12-15T22:58:13.246 INFO:tasks.rados.rados.0.smithi071.stdout:422: done (2 left) 2023-12-15T22:58:13.246 INFO:tasks.rados.rados.0.smithi071.stdout:423: done (1 left) 2023-12-15T22:58:13.246 INFO:tasks.rados.rados.0.smithi071.stdout:425: done (0 left) 2023-12-15T22:58:13.246 INFO:tasks.rados.rados.0.smithi071.stdout:426: snap_create 2023-12-15T22:58:13.300 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:13.296+0000 7f590479e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:13.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:13.303+0000 7f590479e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:13.316 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:58:13.316 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:58:13.316 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:58:13.316 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:58:13.316 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:58:13.316 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:58:13.317 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:58:13.317 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:58:13.317 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:58:13.317 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:58:13.317 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:58:13.317 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:58:13.319 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:58:13.319 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:58:14.243 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:58:14.248 INFO:tasks.rados.rados.0.smithi071.stdout:426: done (0 left) 2023-12-15T22:58:14.248 INFO:tasks.rados.rados.0.smithi071.stdout:427: write (excl) oid 14 current snap is 49 2023-12-15T22:58:14.248 INFO:tasks.rados.rados.0.smithi071.stdout:427: seq_num 219 ranges {480733=537415,1782380=696402,3031400=537159} 2023-12-15T22:58:14.254 INFO:tasks.rados.rados.0.smithi071.stdout:427: writing smithi07183189-14 from 480733 to 1018148 tid 1 2023-12-15T22:58:14.262 INFO:tasks.rados.rados.0.smithi071.stdout:427: writing smithi07183189-14 from 1782380 to 2478782 tid 2 2023-12-15T22:58:14.268 INFO:tasks.rados.rados.0.smithi071.stdout:427: writing smithi07183189-14 from 3031400 to 3568559 tid 3 2023-12-15T22:58:14.268 INFO:tasks.rados.rados.0.smithi071.stdout:428: write (excl) oid 17 current snap is 49 2023-12-15T22:58:14.268 INFO:tasks.rados.rados.0.smithi071.stdout:428: seq_num 220 ranges {497540=612664,1840862=440060,3058823=464784} 2023-12-15T22:58:14.274 INFO:tasks.rados.rados.0.smithi071.stdout:428: writing smithi07183189-17 from 497540 to 1110204 tid 1 2023-12-15T22:58:14.281 INFO:tasks.rados.rados.0.smithi071.stdout:428: writing smithi07183189-17 from 1840862 to 2280922 tid 2 2023-12-15T22:58:14.289 INFO:tasks.rados.rados.0.smithi071.stdout:428: writing smithi07183189-17 from 3058823 to 3523607 tid 3 2023-12-15T22:58:14.289 INFO:tasks.rados.rados.0.smithi071.stdout:429: write oid 24 current snap is 49 2023-12-15T22:58:14.289 INFO:tasks.rados.rados.0.smithi071.stdout:429: seq_num 221 ranges {514347=404266,1499344=700071,2686246=676056} 2023-12-15T22:58:14.294 INFO:tasks.rados.rados.0.smithi071.stdout:429: writing smithi07183189-24 from 514347 to 918613 tid 1 2023-12-15T22:58:14.301 INFO:tasks.rados.rados.0.smithi071.stdout:429: writing smithi07183189-24 from 1499344 to 2199415 tid 2 2023-12-15T22:58:14.308 INFO:tasks.rados.rados.0.smithi071.stdout:429: writing smithi07183189-24 from 2686246 to 3362302 tid 3 2023-12-15T22:58:14.308 INFO:tasks.rados.rados.0.smithi071.stdout:430: snap_remove snap 41 2023-12-15T22:58:14.369 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:14.367+0000 7ff85df9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:14.372 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:14.372+0000 7ff85df9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:15.255 INFO:tasks.rados.rados.0.smithi071.stdout:430: done (3 left) 2023-12-15T22:58:15.255 INFO:tasks.rados.rados.0.smithi071.stdout:431: snap_create 2023-12-15T22:58:15.255 INFO:tasks.rados.rados.0.smithi071.stdout:427: finishing write tid 1 to smithi07183189-14 2023-12-15T22:58:15.255 INFO:tasks.rados.rados.0.smithi071.stdout:428: finishing write tid 1 to smithi07183189-17 2023-12-15T22:58:15.255 INFO:tasks.rados.rados.0.smithi071.stdout:428: finishing write tid 2 to smithi07183189-17 2023-12-15T22:58:15.255 INFO:tasks.rados.rados.0.smithi071.stdout:429: finishing write tid 1 to smithi07183189-24 2023-12-15T22:58:15.255 INFO:tasks.rados.rados.0.smithi071.stdout:428: finishing write tid 3 to smithi07183189-17 2023-12-15T22:58:15.255 INFO:tasks.rados.rados.0.smithi071.stdout:427: finishing write tid 2 to smithi07183189-14 2023-12-15T22:58:15.255 INFO:tasks.rados.rados.0.smithi071.stdout:429: finishing write tid 2 to smithi07183189-24 2023-12-15T22:58:15.255 INFO:tasks.rados.rados.0.smithi071.stdout:428: finishing write tid 4 to smithi07183189-17 2023-12-15T22:58:15.256 INFO:tasks.rados.rados.0.smithi071.stdout:428: finishing write tid 5 to smithi07183189-17 2023-12-15T22:58:15.256 INFO:tasks.rados.rados.0.smithi071.stdout:428: oid 17 updating version 0 to 149 2023-12-15T22:58:15.256 INFO:tasks.rados.rados.0.smithi071.stdout:428: oid 17 version 149 is already newer than 146 2023-12-15T22:58:15.256 INFO:tasks.rados.rados.0.smithi071.stdout:428: oid 17 version 149 is already newer than 148 2023-12-15T22:58:15.256 INFO:tasks.rados.rados.0.smithi071.stdout:428: oid 17 version 149 is already newer than 148 2023-12-15T22:58:15.256 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 17 v 149 (ObjNum 220 snap 49 seq_num 220) dirty exists 2023-12-15T22:58:15.256 INFO:tasks.rados.rados.0.smithi071.stdout:428: left oid 17 (ObjNum 220 snap 49 seq_num 220) 2023-12-15T22:58:15.256 INFO:tasks.rados.rados.0.smithi071.stdout:427: finishing write tid 3 to smithi07183189-14 2023-12-15T22:58:15.256 INFO:tasks.rados.rados.0.smithi071.stdout:429: finishing write tid 3 to smithi07183189-24 2023-12-15T22:58:15.256 INFO:tasks.rados.rados.0.smithi071.stdout:427: finishing write tid 4 to smithi07183189-14 2023-12-15T22:58:15.256 INFO:tasks.rados.rados.0.smithi071.stdout:429: finishing write tid 4 to smithi07183189-24 2023-12-15T22:58:15.256 INFO:tasks.rados.rados.0.smithi071.stdout:427: finishing write tid 5 to smithi07183189-14 2023-12-15T22:58:15.256 INFO:tasks.rados.rados.0.smithi071.stdout:427: oid 14 updating version 0 to 106 2023-12-15T22:58:15.256 INFO:tasks.rados.rados.0.smithi071.stdout:427: oid 14 updating version 106 to 107 2023-12-15T22:58:15.256 INFO:tasks.rados.rados.0.smithi071.stdout:427: oid 14 version 107 is already newer than 101 2023-12-15T22:58:15.257 INFO:tasks.rados.rados.0.smithi071.stdout:427: oid 14 updating version 107 to 108 2023-12-15T22:58:15.257 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 108 (ObjNum 219 snap 49 seq_num 219) dirty exists 2023-12-15T22:58:15.257 INFO:tasks.rados.rados.0.smithi071.stdout:427: left oid 14 (ObjNum 219 snap 49 seq_num 219) 2023-12-15T22:58:15.257 INFO:tasks.rados.rados.0.smithi071.stdout:429: finishing write tid 5 to smithi07183189-24 2023-12-15T22:58:15.257 INFO:tasks.rados.rados.0.smithi071.stdout:429: oid 24 updating version 0 to 31 2023-12-15T22:58:15.257 INFO:tasks.rados.rados.0.smithi071.stdout:429: oid 24 version 31 is already newer than 30 2023-12-15T22:58:15.257 INFO:tasks.rados.rados.0.smithi071.stdout:429: oid 24 version 31 is already newer than 28 2023-12-15T22:58:15.257 INFO:tasks.rados.rados.0.smithi071.stdout:429: oid 24 version 31 is already newer than 29 2023-12-15T22:58:15.257 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 31 (ObjNum 221 snap 49 seq_num 221) dirty exists 2023-12-15T22:58:15.257 INFO:tasks.rados.rados.0.smithi071.stdout:429: left oid 24 (ObjNum 221 snap 49 seq_num 221) 2023-12-15T22:58:16.255 INFO:tasks.rados.rados.0.smithi071.stdout:427: done (3 left) 2023-12-15T22:58:16.256 INFO:tasks.rados.rados.0.smithi071.stdout:428: done (2 left) 2023-12-15T22:58:16.256 INFO:tasks.rados.rados.0.smithi071.stdout:429: done (1 left) 2023-12-15T22:58:16.256 INFO:tasks.rados.rados.0.smithi071.stdout:431: done (0 left) 2023-12-15T22:58:16.256 INFO:tasks.rados.rados.0.smithi071.stdout:432: delete oid 12 current snap is 50 2023-12-15T22:58:16.304 INFO:tasks.rados.rados.0.smithi071.stdout:432: done (0 left) 2023-12-15T22:58:16.304 INFO:tasks.rados.rados.0.smithi071.stdout:433: write (excl) oid 43 current snap is 50 2023-12-15T22:58:16.304 INFO:tasks.rados.rados.0.smithi071.stdout:433: seq_num 223 ranges {547961=554764,1616308=703740,2741092=698600} 2023-12-15T22:58:16.309 INFO:tasks.rados.rados.0.smithi071.stdout:433: writing smithi07183189-43 from 547961 to 1102725 tid 1 2023-12-15T22:58:16.316 INFO:tasks.rados.rados.0.smithi071.stdout:433: writing smithi07183189-43 from 1616308 to 2320048 tid 2 2023-12-15T22:58:16.322 INFO:tasks.rados.rados.0.smithi071.stdout:433: writing smithi07183189-43 from 2741092 to 3439692 tid 3 2023-12-15T22:58:16.323 INFO:tasks.rados.rados.0.smithi071.stdout:434: snap_create 2023-12-15T22:58:16.339 INFO:tasks.rados.rados.0.smithi071.stdout:433: finishing write tid 1 to smithi07183189-43 2023-12-15T22:58:16.355 INFO:tasks.rados.rados.0.smithi071.stdout:433: finishing write tid 2 to smithi07183189-43 2023-12-15T22:58:16.368 INFO:tasks.rados.rados.0.smithi071.stdout:433: finishing write tid 3 to smithi07183189-43 2023-12-15T22:58:16.375 INFO:tasks.rados.rados.0.smithi071.stdout:433: finishing write tid 4 to smithi07183189-43 2023-12-15T22:58:16.376 INFO:tasks.rados.rados.0.smithi071.stdout:433: finishing write tid 5 to smithi07183189-43 2023-12-15T22:58:16.376 INFO:tasks.rados.rados.0.smithi071.stdout:433: oid 43 updating version 0 to 28 2023-12-15T22:58:16.376 INFO:tasks.rados.rados.0.smithi071.stdout:433: oid 43 updating version 28 to 29 2023-12-15T22:58:16.376 INFO:tasks.rados.rados.0.smithi071.stdout:433: oid 43 updating version 29 to 30 2023-12-15T22:58:16.376 INFO:tasks.rados.rados.0.smithi071.stdout:433: oid 43 updating version 30 to 31 2023-12-15T22:58:16.376 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 31 (ObjNum 223 snap 50 seq_num 223) dirty exists 2023-12-15T22:58:16.376 INFO:tasks.rados.rados.0.smithi071.stdout:433: left oid 43 (ObjNum 223 snap 50 seq_num 223) 2023-12-15T22:58:16.548 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:58:16.653 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:16.652+0000 7f842e86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:16.656 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:16.656+0000 7f842e86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:17.266 INFO:tasks.rados.rados.0.smithi071.stdout:433: done (1 left) 2023-12-15T22:58:17.267 INFO:tasks.rados.rados.0.smithi071.stdout:434: done (0 left) 2023-12-15T22:58:17.267 INFO:tasks.rados.rados.0.smithi071.stdout:435: snap_create 2023-12-15T22:58:18.268 INFO:tasks.rados.rados.0.smithi071.stdout:435: done (0 left) 2023-12-15T22:58:18.268 INFO:tasks.rados.rados.0.smithi071.stdout:436: write oid 45 current snap is 52 2023-12-15T22:58:18.268 INFO:tasks.rados.rados.0.smithi071.stdout:436: seq_num 224 ranges {564768=630013,1674790=447398,2768515=626225} 2023-12-15T22:58:18.274 INFO:tasks.rados.rados.0.smithi071.stdout:436: writing smithi07183189-45 from 564768 to 1194781 tid 1 2023-12-15T22:58:18.280 INFO:tasks.rados.rados.0.smithi071.stdout:436: writing smithi07183189-45 from 1674790 to 2122188 tid 2 2023-12-15T22:58:18.287 INFO:tasks.rados.rados.0.smithi071.stdout:436: writing smithi07183189-45 from 2768515 to 3394740 tid 3 2023-12-15T22:58:18.287 INFO:tasks.rados.rados.0.smithi071.stdout:437: snap_create 2023-12-15T22:58:18.320 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T22:58:18.320 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T22:58:18.320 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-12-15T22:58:18.320 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-12-15T22:58:18.340 INFO:tasks.rados.rados.0.smithi071.stdout:436: finishing write tid 1 to smithi07183189-45 2023-12-15T22:58:18.341 INFO:tasks.rados.rados.0.smithi071.stdout:436: finishing write tid 2 to smithi07183189-45 2023-12-15T22:58:18.364 INFO:tasks.rados.rados.0.smithi071.stdout:436: finishing write tid 3 to smithi07183189-45 2023-12-15T22:58:18.376 INFO:tasks.rados.rados.0.smithi071.stdout:436: finishing write tid 4 to smithi07183189-45 2023-12-15T22:58:18.377 INFO:tasks.rados.rados.0.smithi071.stdout:436: finishing write tid 5 to smithi07183189-45 2023-12-15T22:58:18.377 INFO:tasks.rados.rados.0.smithi071.stdout:436: oid 45 updating version 0 to 214 2023-12-15T22:58:18.377 INFO:tasks.rados.rados.0.smithi071.stdout:436: oid 45 updating version 214 to 215 2023-12-15T22:58:18.377 INFO:tasks.rados.rados.0.smithi071.stdout:436: oid 45 version 215 is already newer than 214 2023-12-15T22:58:18.377 INFO:tasks.rados.rados.0.smithi071.stdout:436: oid 45 updating version 215 to 216 2023-12-15T22:58:18.377 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 216 (ObjNum 224 snap 52 seq_num 224) dirty exists 2023-12-15T22:58:18.377 INFO:tasks.rados.rados.0.smithi071.stdout:436: left oid 45 (ObjNum 224 snap 52 seq_num 224) 2023-12-15T22:58:18.436 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:18.432+0000 7fac27d08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:18.440 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:18.440+0000 7fac27d08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:18.451 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:58:18.452 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:58:18.452 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:58:18.452 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:58:18.452 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:58:18.452 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:58:18.452 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:58:18.452 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:58:18.452 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:58:18.452 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:58:18.452 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:58:18.452 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:58:18.454 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:58:18.455 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:58:18.455 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-12-15T22:58:18.601 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:18.596+0000 7f906239e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:18.604 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:18.604+0000 7f906239e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:18.616 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:58:18.616 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:58:18.616 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:58:18.616 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:58:18.616 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:58:18.616 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:58:18.617 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:58:18.617 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:58:18.617 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:58:18.617 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:58:18.617 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:58:18.617 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:58:18.619 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:58:18.620 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:58:18.620 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-12-15T22:58:18.724 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:58:18.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:18.721+0000 7f32a426d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:18.728 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:18.728+0000 7f32a426d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:18.742 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:58:18.742 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:58:18.742 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:58:18.742 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:58:18.742 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:58:18.742 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:58:18.742 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:58:18.742 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:58:18.742 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:58:18.742 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:58:18.742 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:58:18.742 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:58:18.745 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:58:18.745 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:58:18.745 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-12-15T22:58:18.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:18.828+0000 7f98d931e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:18.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:18.832+0000 7f98d931e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:18.897 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:18.891+0000 7f454799e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:18.900 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:18.900+0000 7f454799e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:18.913 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:58:18.913 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:58:18.913 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:58:18.913 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:58:18.913 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:58:18.913 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:58:18.913 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:58:18.913 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:58:18.914 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:58:18.914 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:58:18.914 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:58:18.914 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:58:18.915 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:58:18.916 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:58:19.277 INFO:tasks.rados.rados.0.smithi071.stdout:436: done (1 left) 2023-12-15T22:58:19.278 INFO:tasks.rados.rados.0.smithi071.stdout:437: done (0 left) 2023-12-15T22:58:19.278 INFO:tasks.rados.rados.0.smithi071.stdout:438: snap_create 2023-12-15T22:58:20.279 INFO:tasks.rados.rados.0.smithi071.stdout:438: done (0 left) 2023-12-15T22:58:20.279 INFO:tasks.rados.rados.0.smithi071.stdout:439: delete oid 13 current snap is 54 2023-12-15T22:58:20.338 INFO:tasks.rados.rados.0.smithi071.stdout:439: done (0 left) 2023-12-15T22:58:20.338 INFO:tasks.rados.rados.0.smithi071.stdout:440: write (excl) oid 36 current snap is 54 2023-12-15T22:58:20.338 INFO:tasks.rados.rados.0.smithi071.stdout:440: seq_num 226 ranges {598382=780511,2075401=451067,3107008=648769} 2023-12-15T22:58:20.345 INFO:tasks.rados.rados.0.smithi071.stdout:440: writing smithi07183189-36 from 598382 to 1378893 tid 1 2023-12-15T22:58:20.352 INFO:tasks.rados.rados.0.smithi071.stdout:440: writing smithi07183189-36 from 2075401 to 2526468 tid 2 2023-12-15T22:58:20.359 INFO:tasks.rados.rados.0.smithi071.stdout:440: writing smithi07183189-36 from 3107008 to 3755777 tid 3 2023-12-15T22:58:20.359 INFO:tasks.rados.rados.0.smithi071.stdout:441: snap_create 2023-12-15T22:58:20.377 INFO:tasks.rados.rados.0.smithi071.stdout:440: finishing write tid 1 to smithi07183189-36 2023-12-15T22:58:20.397 INFO:tasks.rados.rados.0.smithi071.stdout:440: finishing write tid 2 to smithi07183189-36 2023-12-15T22:58:20.413 INFO:tasks.rados.rados.0.smithi071.stdout:440: finishing write tid 3 to smithi07183189-36 2023-12-15T22:58:20.421 INFO:tasks.rados.rados.0.smithi071.stdout:440: finishing write tid 4 to smithi07183189-36 2023-12-15T22:58:20.423 INFO:tasks.rados.rados.0.smithi071.stdout:440: finishing write tid 5 to smithi07183189-36 2023-12-15T22:58:20.423 INFO:tasks.rados.rados.0.smithi071.stdout:440: oid 36 updating version 0 to 158 2023-12-15T22:58:20.423 INFO:tasks.rados.rados.0.smithi071.stdout:440: oid 36 version 158 is already newer than 155 2023-12-15T22:58:20.423 INFO:tasks.rados.rados.0.smithi071.stdout:440: oid 36 version 158 is already newer than 156 2023-12-15T22:58:20.423 INFO:tasks.rados.rados.0.smithi071.stdout:440: oid 36 version 158 is already newer than 157 2023-12-15T22:58:20.423 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 158 (ObjNum 226 snap 54 seq_num 226) dirty exists 2023-12-15T22:58:20.423 INFO:tasks.rados.rados.0.smithi071.stdout:440: left oid 36 (ObjNum 226 snap 54 seq_num 226) 2023-12-15T22:58:20.905 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:58:21.014 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:21.013+0000 7f49a5f00640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:21.017 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:21.017+0000 7f49a5f00640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:21.286 INFO:tasks.rados.rados.0.smithi071.stdout:440: done (1 left) 2023-12-15T22:58:21.287 INFO:tasks.rados.rados.0.smithi071.stdout:441: done (0 left) 2023-12-15T22:58:21.287 INFO:tasks.rados.rados.0.smithi071.stdout:442: read oid 48 snap -1 2023-12-15T22:58:21.287 INFO:tasks.rados.rados.0.smithi071.stdout:442: expect (ObjNum 217 snap 47 seq_num 217) 2023-12-15T22:58:21.287 INFO:tasks.rados.rados.0.smithi071.stdout:443: read oid 43 snap 42 2023-12-15T22:58:21.287 INFO:tasks.rados.rados.0.smithi071.stdout:443: expect (ObjNum 73 snap 3 seq_num 73) 2023-12-15T22:58:21.287 INFO:tasks.rados.rados.0.smithi071.stdout:444: write oid 50 current snap is 55 2023-12-15T22:58:21.287 INFO:tasks.rados.rados.0.smithi071.stdout:444: seq_num 227 ranges {615189=455760,1733883=711078,3134431=460041} 2023-12-15T22:58:21.292 INFO:tasks.rados.rados.0.smithi071.stdout:444: writing smithi07183189-50 from 615189 to 1070949 tid 1 2023-12-15T22:58:21.301 INFO:tasks.rados.rados.0.smithi071.stdout:444: writing smithi07183189-50 from 1733883 to 2444961 tid 2 2023-12-15T22:58:21.307 INFO:tasks.rados.rados.0.smithi071.stdout:444: writing smithi07183189-50 from 3134431 to 3594472 tid 3 2023-12-15T22:58:21.308 INFO:tasks.rados.rados.0.smithi071.stdout:445: read oid 1 snap -1 2023-12-15T22:58:21.308 INFO:tasks.rados.rados.0.smithi071.stdout:445: expect (ObjNum 0 snap 0 seq_num 0) 2023-12-15T22:58:21.308 INFO:tasks.rados.rados.0.smithi071.stdout:446: write oid 29 current snap is 55 2023-12-15T22:58:21.308 INFO:tasks.rados.rados.0.smithi071.stdout:446: seq_num 228 ranges {631996=531009,1792365=454736,2761854=671313} 2023-12-15T22:58:21.314 INFO:tasks.rados.rados.0.smithi071.stdout:446: writing smithi07183189-29 from 631996 to 1163005 tid 1 2023-12-15T22:58:21.320 INFO:tasks.rados.rados.0.smithi071.stdout:446: writing smithi07183189-29 from 1792365 to 2247101 tid 2 2023-12-15T22:58:21.327 INFO:tasks.rados.rados.0.smithi071.stdout:446: writing smithi07183189-29 from 2761854 to 3433167 tid 3 2023-12-15T22:58:21.327 INFO:tasks.rados.rados.0.smithi071.stdout:447: snap_remove snap 49 2023-12-15T22:58:22.246 INFO:tasks.rados.rados.0.smithi071.stdout:447: done (5 left) 2023-12-15T22:58:22.246 INFO:tasks.rados.rados.0.smithi071.stdout:448: delete oid 12 current snap is 55 2023-12-15T22:58:22.246 INFO:tasks.rados.rados.0.smithi071.stdout:443: expect (ObjNum 73 snap 3 seq_num 73) 2023-12-15T22:58:22.415 INFO:tasks.rados.rados.0.smithi071.stdout:442: expect (ObjNum 217 snap 47 seq_num 217) 2023-12-15T22:58:22.629 INFO:tasks.rados.rados.0.smithi071.stdout:445: expect (ObjNum 0 snap 0 seq_num 0) 2023-12-15T22:58:22.709 INFO:tasks.rados.rados.0.smithi071.stdout:444: finishing write tid 1 to smithi07183189-50 2023-12-15T22:58:22.709 INFO:tasks.rados.rados.0.smithi071.stdout:444: finishing write tid 2 to smithi07183189-50 2023-12-15T22:58:22.710 INFO:tasks.rados.rados.0.smithi071.stdout:444: finishing write tid 3 to smithi07183189-50 2023-12-15T22:58:22.710 INFO:tasks.rados.rados.0.smithi071.stdout:444: finishing write tid 4 to smithi07183189-50 2023-12-15T22:58:22.710 INFO:tasks.rados.rados.0.smithi071.stdout:442: done (5 left) 2023-12-15T22:58:22.710 INFO:tasks.rados.rados.0.smithi071.stdout:443: done (4 left) 2023-12-15T22:58:22.710 INFO:tasks.rados.rados.0.smithi071.stdout:445: done (3 left) 2023-12-15T22:58:22.710 INFO:tasks.rados.rados.0.smithi071.stdout:448: done (2 left) 2023-12-15T22:58:22.710 INFO:tasks.rados.rados.0.smithi071.stdout:449: read oid 10 snap 47 2023-12-15T22:58:22.710 INFO:tasks.rados.rados.0.smithi071.stdout:449: expect (ObjNum 147 snap 20 seq_num 147) 2023-12-15T22:58:22.710 INFO:tasks.rados.rados.0.smithi071.stdout:450: read oid 18 snap -1 2023-12-15T22:58:22.710 INFO:tasks.rados.rados.0.smithi071.stdout:450: expect (ObjNum 185 snap 34 seq_num 185) 2023-12-15T22:58:22.710 INFO:tasks.rados.rados.0.smithi071.stdout:451: write (excl) oid 27 current snap is 55 2023-12-15T22:58:22.710 INFO:tasks.rados.rados.0.smithi071.stdout:444: finishing write tid 5 to smithi07183189-50 2023-12-15T22:58:22.710 INFO:tasks.rados.rados.0.smithi071.stdout:444: oid 50 updating version 0 to 218 2023-12-15T22:58:22.710 INFO:tasks.rados.rados.0.smithi071.stdout:444: oid 50 updating version 218 to 220 2023-12-15T22:58:22.711 INFO:tasks.rados.rados.0.smithi071.stdout:444: oid 50 version 220 is already newer than 219 2023-12-15T22:58:22.711 INFO:tasks.rados.rados.0.smithi071.stdout:444: oid 50 updating version 220 to 221 2023-12-15T22:58:22.711 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 221 (ObjNum 227 snap 55 seq_num 227) dirty exists 2023-12-15T22:58:22.711 INFO:tasks.rados.rados.0.smithi071.stdout:444: left oid 50 (ObjNum 227 snap 55 seq_num 227) 2023-12-15T22:58:22.711 INFO:tasks.rados.rados.0.smithi071.stdout:446: finishing write tid 1 to smithi07183189-29 2023-12-15T22:58:22.711 INFO:tasks.rados.rados.0.smithi071.stdout:451: seq_num 230 ranges {665610=797860,1909329=458405,2816700=410210} 2023-12-15T22:58:22.717 INFO:tasks.rados.rados.0.smithi071.stdout:451: writing smithi07183189-27 from 665610 to 1463470 tid 1 2023-12-15T22:58:22.722 INFO:tasks.rados.rados.0.smithi071.stdout:451: writing smithi07183189-27 from 1909329 to 2367734 tid 2 2023-12-15T22:58:22.728 INFO:tasks.rados.rados.0.smithi071.stdout:451: writing smithi07183189-27 from 2816700 to 3226910 tid 3 2023-12-15T22:58:22.728 INFO:tasks.rados.rados.0.smithi071.stdout:444: done (4 left) 2023-12-15T22:58:22.728 INFO:tasks.rados.rados.0.smithi071.stdout:452: read oid 34 snap 47 2023-12-15T22:58:22.728 INFO:tasks.rados.rados.0.smithi071.stdout:452: expect (ObjNum 196 snap 40 seq_num 196) 2023-12-15T22:58:22.728 INFO:tasks.rados.rados.0.smithi071.stdout:453: write (excl) oid 6 current snap is 55 2023-12-15T22:58:22.728 INFO:tasks.rados.rados.0.smithi071.stdout:446: finishing write tid 2 to smithi07183189-29 2023-12-15T22:58:22.728 INFO:tasks.rados.rados.0.smithi071.stdout:446: finishing write tid 3 to smithi07183189-29 2023-12-15T22:58:22.728 INFO:tasks.rados.rados.0.smithi071.stdout:446: finishing write tid 4 to smithi07183189-29 2023-12-15T22:58:22.728 INFO:tasks.rados.rados.0.smithi071.stdout:446: finishing write tid 5 to smithi07183189-29 2023-12-15T22:58:22.728 INFO:tasks.rados.rados.0.smithi071.stdout:446: oid 29 updating version 0 to 224 2023-12-15T22:58:22.728 INFO:tasks.rados.rados.0.smithi071.stdout:446: oid 29 updating version 224 to 225 2023-12-15T22:58:22.728 INFO:tasks.rados.rados.0.smithi071.stdout:446: oid 29 version 225 is already newer than 223 2023-12-15T22:58:22.728 INFO:tasks.rados.rados.0.smithi071.stdout:446: oid 29 version 225 is already newer than 223 2023-12-15T22:58:22.728 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 225 (ObjNum 228 snap 55 seq_num 228) dirty exists 2023-12-15T22:58:22.728 INFO:tasks.rados.rados.0.smithi071.stdout:446: left oid 29 (ObjNum 228 snap 55 seq_num 228) 2023-12-15T22:58:22.729 INFO:tasks.rados.rados.0.smithi071.stdout:453: seq_num 231 ranges {682417=473109,1567811=602063,2844123=621482} 2023-12-15T22:58:22.733 INFO:tasks.rados.rados.0.smithi071.stdout:453: writing smithi07183189-6 from 682417 to 1155526 tid 1 2023-12-15T22:58:22.739 INFO:tasks.rados.rados.0.smithi071.stdout:453: writing smithi07183189-6 from 1567811 to 2169874 tid 2 2023-12-15T22:58:22.747 INFO:tasks.rados.rados.0.smithi071.stdout:453: writing smithi07183189-6 from 2844123 to 3465605 tid 3 2023-12-15T22:58:22.747 INFO:tasks.rados.rados.0.smithi071.stdout:446: done (5 left) 2023-12-15T22:58:22.747 INFO:tasks.rados.rados.0.smithi071.stdout:454: write oid 14 current snap is 55 2023-12-15T22:58:22.747 INFO:tasks.rados.rados.0.smithi071.stdout:454: seq_num 232 ranges {699224=548358,2026293=462074,3271546=432754} 2023-12-15T22:58:22.753 INFO:tasks.rados.rados.0.smithi071.stdout:454: writing smithi07183189-14 from 699224 to 1247582 tid 1 2023-12-15T22:58:22.760 INFO:tasks.rados.rados.0.smithi071.stdout:454: writing smithi07183189-14 from 2026293 to 2488367 tid 2 2023-12-15T22:58:22.767 INFO:tasks.rados.rados.0.smithi071.stdout:454: writing smithi07183189-14 from 3271546 to 3704300 tid 3 2023-12-15T22:58:22.767 INFO:tasks.rados.rados.0.smithi071.stdout:455: snap_create 2023-12-15T22:58:22.767 INFO:tasks.rados.rados.0.smithi071.stdout:449: expect (ObjNum 147 snap 20 seq_num 147) 2023-12-15T22:58:22.881 INFO:tasks.rados.rados.0.smithi071.stdout:450: expect (ObjNum 185 snap 34 seq_num 185) 2023-12-15T22:58:23.044 INFO:tasks.rados.rados.0.smithi071.stdout:452: expect (ObjNum 196 snap 40 seq_num 196) 2023-12-15T22:58:23.088 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:58:23.193 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:23.190+0000 7f8b9a95e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:23.195 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:23.194+0000 7f8b9a95e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:23.210 INFO:tasks.rados.rados.0.smithi071.stdout:451: finishing write tid 1 to smithi07183189-27 2023-12-15T22:58:23.210 INFO:tasks.rados.rados.0.smithi071.stdout:453: finishing write tid 1 to smithi07183189-6 2023-12-15T22:58:23.210 INFO:tasks.rados.rados.0.smithi071.stdout:454: finishing write tid 1 to smithi07183189-14 2023-12-15T22:58:23.210 INFO:tasks.rados.rados.0.smithi071.stdout:451: finishing write tid 2 to smithi07183189-27 2023-12-15T22:58:23.210 INFO:tasks.rados.rados.0.smithi071.stdout:453: finishing write tid 2 to smithi07183189-6 2023-12-15T22:58:23.210 INFO:tasks.rados.rados.0.smithi071.stdout:451: finishing write tid 3 to smithi07183189-27 2023-12-15T22:58:23.210 INFO:tasks.rados.rados.0.smithi071.stdout:454: finishing write tid 2 to smithi07183189-14 2023-12-15T22:58:23.210 INFO:tasks.rados.rados.0.smithi071.stdout:451: finishing write tid 4 to smithi07183189-27 2023-12-15T22:58:23.211 INFO:tasks.rados.rados.0.smithi071.stdout:451: finishing write tid 5 to smithi07183189-27 2023-12-15T22:58:23.211 INFO:tasks.rados.rados.0.smithi071.stdout:451: oid 27 updating version 0 to 80 2023-12-15T22:58:23.211 INFO:tasks.rados.rados.0.smithi071.stdout:451: oid 27 version 80 is already newer than 80 2023-12-15T22:58:23.211 INFO:tasks.rados.rados.0.smithi071.stdout:451: oid 27 version 80 is already newer than 78 2023-12-15T22:58:23.211 INFO:tasks.rados.rados.0.smithi071.stdout:451: oid 27 updating version 80 to 81 2023-12-15T22:58:23.211 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 81 (ObjNum 230 snap 55 seq_num 230) dirty exists 2023-12-15T22:58:23.211 INFO:tasks.rados.rados.0.smithi071.stdout:451: left oid 27 (ObjNum 230 snap 55 seq_num 230) 2023-12-15T22:58:23.211 INFO:tasks.rados.rados.0.smithi071.stdout:454: finishing write tid 3 to smithi07183189-14 2023-12-15T22:58:23.211 INFO:tasks.rados.rados.0.smithi071.stdout:453: finishing write tid 3 to smithi07183189-6 2023-12-15T22:58:23.211 INFO:tasks.rados.rados.0.smithi071.stdout:453: finishing write tid 4 to smithi07183189-6 2023-12-15T22:58:23.211 INFO:tasks.rados.rados.0.smithi071.stdout:453: finishing write tid 5 to smithi07183189-6 2023-12-15T22:58:23.211 INFO:tasks.rados.rados.0.smithi071.stdout:453: oid 6 updating version 0 to 51 2023-12-15T22:58:23.211 INFO:tasks.rados.rados.0.smithi071.stdout:453: oid 6 version 51 is already newer than 49 2023-12-15T22:58:23.211 INFO:tasks.rados.rados.0.smithi071.stdout:453: oid 6 version 51 is already newer than 48 2023-12-15T22:58:23.211 INFO:tasks.rados.rados.0.smithi071.stdout:453: oid 6 version 51 is already newer than 51 2023-12-15T22:58:23.212 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 6 v 51 (ObjNum 231 snap 55 seq_num 231) dirty exists 2023-12-15T22:58:23.212 INFO:tasks.rados.rados.0.smithi071.stdout:453: left oid 6 (ObjNum 231 snap 55 seq_num 231) 2023-12-15T22:58:23.212 INFO:tasks.rados.rados.0.smithi071.stdout:454: finishing write tid 4 to smithi07183189-14 2023-12-15T22:58:23.212 INFO:tasks.rados.rados.0.smithi071.stdout:454: finishing write tid 5 to smithi07183189-14 2023-12-15T22:58:23.212 INFO:tasks.rados.rados.0.smithi071.stdout:454: oid 14 updating version 0 to 120 2023-12-15T22:58:23.212 INFO:tasks.rados.rados.0.smithi071.stdout:454: oid 14 updating version 120 to 121 2023-12-15T22:58:23.212 INFO:tasks.rados.rados.0.smithi071.stdout:454: oid 14 version 121 is already newer than 119 2023-12-15T22:58:23.212 INFO:tasks.rados.rados.0.smithi071.stdout:454: oid 14 version 121 is already newer than 118 2023-12-15T22:58:23.212 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 121 (ObjNum 232 snap 55 seq_num 232) dirty exists 2023-12-15T22:58:23.212 INFO:tasks.rados.rados.0.smithi071.stdout:454: left oid 14 (ObjNum 232 snap 55 seq_num 232) 2023-12-15T22:58:23.290 INFO:tasks.rados.rados.0.smithi071.stdout:449: done (6 left) 2023-12-15T22:58:23.290 INFO:tasks.rados.rados.0.smithi071.stdout:450: done (5 left) 2023-12-15T22:58:23.290 INFO:tasks.rados.rados.0.smithi071.stdout:451: done (4 left) 2023-12-15T22:58:23.290 INFO:tasks.rados.rados.0.smithi071.stdout:452: done (3 left) 2023-12-15T22:58:23.290 INFO:tasks.rados.rados.0.smithi071.stdout:453: done (2 left) 2023-12-15T22:58:23.290 INFO:tasks.rados.rados.0.smithi071.stdout:454: done (1 left) 2023-12-15T22:58:23.290 INFO:tasks.rados.rados.0.smithi071.stdout:455: done (0 left) 2023-12-15T22:58:23.290 INFO:tasks.rados.rados.0.smithi071.stdout:456: read oid 33 snap 55 2023-12-15T22:58:23.290 INFO:tasks.rados.rados.0.smithi071.stdout:456: expect (ObjNum 200 snap 41 seq_num 200) 2023-12-15T22:58:23.291 INFO:tasks.rados.rados.0.smithi071.stdout:457: snap_create 2023-12-15T22:58:23.336 INFO:tasks.rados.rados.0.smithi071.stdout:456: expect (ObjNum 200 snap 41 seq_num 200) 2023-12-15T22:58:23.917 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T22:58:23.917 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T22:58:23.917 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-12-15T22:58:23.917 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-12-15T22:58:24.026 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:24.020+0000 7f314279e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:24.029 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:24.028+0000 7f314279e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:24.041 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:58:24.041 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:58:24.041 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:58:24.041 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:58:24.041 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:58:24.041 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:58:24.041 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:58:24.041 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:58:24.041 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:58:24.041 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:58:24.041 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:58:24.042 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:58:24.043 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:58:24.044 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:58:24.044 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-12-15T22:58:24.150 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:24.145+0000 7f1c1c417640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:24.153 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:24.152+0000 7f1c1c417640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:24.166 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:58:24.166 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:58:24.166 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:58:24.166 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:58:24.166 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:58:24.166 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:58:24.166 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:58:24.166 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:58:24.166 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:58:24.166 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:58:24.166 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:58:24.166 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:58:24.168 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:58:24.169 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:58:24.169 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-12-15T22:58:24.275 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:24.270+0000 7f20a8e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:24.278 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:24.277+0000 7f20a8e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:24.291 INFO:tasks.rados.rados.0.smithi071.stdout:456: done (1 left) 2023-12-15T22:58:24.291 INFO:tasks.rados.rados.0.smithi071.stdout:457: done (0 left) 2023-12-15T22:58:24.291 INFO:tasks.rados.rados.0.smithi071.stdout:458: write (excl) oid 45 current snap is 57 2023-12-15T22:58:24.291 INFO:tasks.rados.rados.0.smithi071.stdout:458: seq_num 233 ranges {716031=623607,2084775=605732,3298969=617062} 2023-12-15T22:58:24.293 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:58:24.293 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:58:24.293 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:58:24.293 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:58:24.293 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:58:24.293 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:58:24.293 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:58:24.293 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:58:24.293 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:58:24.293 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:58:24.293 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:58:24.293 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:58:24.296 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:58:24.296 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:58:24.296 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-12-15T22:58:24.299 INFO:tasks.rados.rados.0.smithi071.stdout:458: writing smithi07183189-45 from 716031 to 1339638 tid 1 2023-12-15T22:58:24.307 INFO:tasks.rados.rados.0.smithi071.stdout:458: writing smithi07183189-45 from 2084775 to 2690507 tid 2 2023-12-15T22:58:24.314 INFO:tasks.rados.rados.0.smithi071.stdout:458: writing smithi07183189-45 from 3298969 to 3916031 tid 3 2023-12-15T22:58:24.314 INFO:tasks.rados.rados.0.smithi071.stdout:459: snap_create 2023-12-15T22:58:24.340 INFO:tasks.rados.rados.0.smithi071.stdout:458: finishing write tid 1 to smithi07183189-45 2023-12-15T22:58:24.360 INFO:tasks.rados.rados.0.smithi071.stdout:458: finishing write tid 2 to smithi07183189-45 2023-12-15T22:58:24.380 INFO:tasks.rados.rados.0.smithi071.stdout:458: finishing write tid 3 to smithi07183189-45 2023-12-15T22:58:24.392 INFO:tasks.rados.rados.0.smithi071.stdout:458: finishing write tid 4 to smithi07183189-45 2023-12-15T22:58:24.394 INFO:tasks.rados.rados.0.smithi071.stdout:458: finishing write tid 5 to smithi07183189-45 2023-12-15T22:58:24.394 INFO:tasks.rados.rados.0.smithi071.stdout:458: oid 45 updating version 0 to 234 2023-12-15T22:58:24.394 INFO:tasks.rados.rados.0.smithi071.stdout:458: oid 45 version 234 is already newer than 232 2023-12-15T22:58:24.394 INFO:tasks.rados.rados.0.smithi071.stdout:458: oid 45 version 234 is already newer than 231 2023-12-15T22:58:24.394 INFO:tasks.rados.rados.0.smithi071.stdout:458: oid 45 version 234 is already newer than 233 2023-12-15T22:58:24.394 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 234 (ObjNum 233 snap 57 seq_num 233) dirty exists 2023-12-15T22:58:24.395 INFO:tasks.rados.rados.0.smithi071.stdout:458: left oid 45 (ObjNum 233 snap 57 seq_num 233) 2023-12-15T22:58:24.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:24.415+0000 7fb78cd5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:24.424 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:24.423+0000 7fb78cd5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:24.436 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:58:24.437 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:58:24.437 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:58:24.437 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:58:24.437 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:58:24.437 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:58:24.437 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:58:24.437 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:58:24.437 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:58:24.437 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:58:24.437 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:58:24.437 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:58:24.439 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:58:24.440 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:58:25.263 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:58:25.298 INFO:tasks.rados.rados.0.smithi071.stdout:458: done (1 left) 2023-12-15T22:58:25.298 INFO:tasks.rados.rados.0.smithi071.stdout:459: done (0 left) 2023-12-15T22:58:25.298 INFO:tasks.rados.rados.0.smithi071.stdout:460: read oid 35 snap 52 2023-12-15T22:58:25.298 INFO:tasks.rados.rados.0.smithi071.stdout:460: expect (ObjNum 188 snap 36 seq_num 188) 2023-12-15T22:58:25.298 INFO:tasks.rados.rados.0.smithi071.stdout:461: read oid 2 snap -1 2023-12-15T22:58:25.298 INFO:tasks.rados.rados.0.smithi071.stdout:461: expect deleted 2023-12-15T22:58:25.299 INFO:tasks.rados.rados.0.smithi071.stdout:462: snap_remove snap 48 2023-12-15T22:58:25.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:25.387+0000 7f8ea566d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:25.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:25.391+0000 7f8ea566d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:26.300 INFO:tasks.rados.rados.0.smithi071.stdout:462: done (2 left) 2023-12-15T22:58:26.300 INFO:tasks.rados.rados.0.smithi071.stdout:463: read oid 9 snap -1 2023-12-15T22:58:26.300 INFO:tasks.rados.rados.0.smithi071.stdout:463: expect (ObjNum 170 snap 26 seq_num 170) 2023-12-15T22:58:26.301 INFO:tasks.rados.rados.0.smithi071.stdout:464: read oid 40 snap -1 2023-12-15T22:58:26.301 INFO:tasks.rados.rados.0.smithi071.stdout:464: expect (ObjNum 122 snap 14 seq_num 122) 2023-12-15T22:58:26.301 INFO:tasks.rados.rados.0.smithi071.stdout:465: write (excl) oid 24 current snap is 58 2023-12-15T22:58:26.301 INFO:tasks.rados.rados.0.smithi071.stdout:465: seq_num 234 ranges {732838=698856,2026904=465743,3210039=571651} 2023-12-15T22:58:26.308 INFO:tasks.rados.rados.0.smithi071.stdout:465: writing smithi07183189-24 from 732838 to 1431694 tid 1 2023-12-15T22:58:26.314 INFO:tasks.rados.rados.0.smithi071.stdout:465: writing smithi07183189-24 from 2026904 to 2492647 tid 2 2023-12-15T22:58:26.321 INFO:tasks.rados.rados.0.smithi071.stdout:465: writing smithi07183189-24 from 3210039 to 3781690 tid 3 2023-12-15T22:58:26.321 INFO:tasks.rados.rados.0.smithi071.stdout:466: read oid 17 snap -1 2023-12-15T22:58:26.321 INFO:tasks.rados.rados.0.smithi071.stdout:466: expect (ObjNum 220 snap 49 seq_num 220) 2023-12-15T22:58:26.321 INFO:tasks.rados.rados.0.smithi071.stdout:467: snap_remove snap 32 2023-12-15T22:58:27.312 INFO:tasks.rados.rados.0.smithi071.stdout:467: done (6 left) 2023-12-15T22:58:27.312 INFO:tasks.rados.rados.0.smithi071.stdout:468: delete oid 20 current snap is 58 2023-12-15T22:58:27.312 INFO:tasks.rados.rados.0.smithi071.stdout:460: expect (ObjNum 188 snap 36 seq_num 188) 2023-12-15T22:58:27.462 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:58:27.567 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:27.566+0000 7f276bef0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:27.605 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:27.570+0000 7f276bef0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:27.606 INFO:tasks.rados.rados.0.smithi071.stdout:464: expect (ObjNum 122 snap 14 seq_num 122) 2023-12-15T22:58:27.672 INFO:tasks.rados.rados.0.smithi071.stdout:463: expect (ObjNum 170 snap 26 seq_num 170) 2023-12-15T22:58:27.860 INFO:tasks.rados.rados.0.smithi071.stdout:465: finishing write tid 1 to smithi07183189-24 2023-12-15T22:58:27.861 INFO:tasks.rados.rados.0.smithi071.stdout:465: finishing write tid 2 to smithi07183189-24 2023-12-15T22:58:27.861 INFO:tasks.rados.rados.0.smithi071.stdout:466: expect (ObjNum 220 snap 49 seq_num 220) 2023-12-15T22:58:28.135 INFO:tasks.rados.rados.0.smithi071.stdout:465: finishing write tid 3 to smithi07183189-24 2023-12-15T22:58:28.135 INFO:tasks.rados.rados.0.smithi071.stdout:465: finishing write tid 4 to smithi07183189-24 2023-12-15T22:58:28.135 INFO:tasks.rados.rados.0.smithi071.stdout:465: finishing write tid 5 to smithi07183189-24 2023-12-15T22:58:28.135 INFO:tasks.rados.rados.0.smithi071.stdout:465: oid 24 updating version 0 to 35 2023-12-15T22:58:28.135 INFO:tasks.rados.rados.0.smithi071.stdout:465: oid 24 updating version 35 to 36 2023-12-15T22:58:28.135 INFO:tasks.rados.rados.0.smithi071.stdout:465: oid 24 updating version 36 to 39 2023-12-15T22:58:28.135 INFO:tasks.rados.rados.0.smithi071.stdout:465: oid 24 updating version 39 to 40 2023-12-15T22:58:28.135 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 40 (ObjNum 234 snap 58 seq_num 234) dirty exists 2023-12-15T22:58:28.135 INFO:tasks.rados.rados.0.smithi071.stdout:465: left oid 24 (ObjNum 234 snap 58 seq_num 234) 2023-12-15T22:58:28.135 INFO:tasks.rados.rados.0.smithi071.stdout:460: done (6 left) 2023-12-15T22:58:28.136 INFO:tasks.rados.rados.0.smithi071.stdout:461: done (5 left) 2023-12-15T22:58:28.136 INFO:tasks.rados.rados.0.smithi071.stdout:463: done (4 left) 2023-12-15T22:58:28.136 INFO:tasks.rados.rados.0.smithi071.stdout:464: done (3 left) 2023-12-15T22:58:28.136 INFO:tasks.rados.rados.0.smithi071.stdout:465: done (2 left) 2023-12-15T22:58:28.136 INFO:tasks.rados.rados.0.smithi071.stdout:466: done (1 left) 2023-12-15T22:58:28.136 INFO:tasks.rados.rados.0.smithi071.stdout:468: done (0 left) 2023-12-15T22:58:28.136 INFO:tasks.rados.rados.0.smithi071.stdout:469: read oid 15 snap 55 2023-12-15T22:58:28.136 INFO:tasks.rados.rados.0.smithi071.stdout:469: expect (ObjNum 197 snap 40 seq_num 197) 2023-12-15T22:58:28.136 INFO:tasks.rados.rados.0.smithi071.stdout:470: snap_create 2023-12-15T22:58:28.145 INFO:tasks.rados.rados.0.smithi071.stdout:469: expect (ObjNum 197 snap 40 seq_num 197) 2023-12-15T22:58:28.313 INFO:tasks.rados.rados.0.smithi071.stdout:469: done (1 left) 2023-12-15T22:58:28.313 INFO:tasks.rados.rados.0.smithi071.stdout:470: done (0 left) 2023-12-15T22:58:28.313 INFO:tasks.rados.rados.0.smithi071.stdout:471: delete oid 12 current snap is 59 2023-12-15T22:58:28.365 INFO:tasks.rados.rados.0.smithi071.stdout:471: done (0 left) 2023-12-15T22:58:28.366 INFO:tasks.rados.rados.0.smithi071.stdout:472: snap_remove snap 52 2023-12-15T22:58:29.325 INFO:tasks.rados.rados.0.smithi071.stdout:472: done (0 left) 2023-12-15T22:58:29.325 INFO:tasks.rados.rados.0.smithi071.stdout:473: snap_create 2023-12-15T22:58:29.441 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T22:58:29.441 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T22:58:29.441 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-12-15T22:58:29.442 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-12-15T22:58:29.561 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:29.556+0000 7fcec8b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:29.564 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:29.564+0000 7fcec8b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:29.577 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:58:29.577 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:58:29.577 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:58:29.577 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:58:29.577 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:58:29.577 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:58:29.577 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:58:29.577 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:58:29.577 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:58:29.577 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:58:29.577 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:58:29.577 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:58:29.579 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:58:29.580 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:58:29.580 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-12-15T22:58:29.655 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:58:29.688 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:29.683+0000 7fb63879e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:29.693 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:29.691+0000 7fb63879e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:29.711 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:58:29.712 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:58:29.712 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:58:29.712 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:58:29.712 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:58:29.712 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:58:29.712 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:58:29.712 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:58:29.712 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:58:29.712 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:58:29.712 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:58:29.712 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:58:29.715 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:58:29.715 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:58:29.715 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-12-15T22:58:29.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:29.765+0000 7f61484ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:29.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:29.769+0000 7f61484ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:29.866 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:29.862+0000 7f23f422d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:29.869 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:29.870+0000 7f23f422d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:29.882 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:58:29.882 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:58:29.882 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:58:29.882 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:58:29.882 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:58:29.882 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:58:29.882 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:58:29.882 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:58:29.882 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:58:29.882 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:58:29.882 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:58:29.882 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:58:29.884 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:58:29.885 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:58:29.885 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-12-15T22:58:30.031 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:30.026+0000 7fc8bc19e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:30.034 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:30.034+0000 7fc8bc19e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:30.047 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:58:30.047 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:58:30.048 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:58:30.048 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:58:30.048 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:58:30.048 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:58:30.048 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:58:30.048 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:58:30.048 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:58:30.048 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:58:30.048 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:58:30.048 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:58:30.050 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:58:30.050 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:58:30.327 INFO:tasks.rados.rados.0.smithi071.stdout:473: done (0 left) 2023-12-15T22:58:30.327 INFO:tasks.rados.rados.0.smithi071.stdout:474: snap_remove snap 47 2023-12-15T22:58:31.326 INFO:tasks.rados.rados.0.smithi071.stdout:474: done (0 left) 2023-12-15T22:58:31.326 INFO:tasks.rados.rados.0.smithi071.stdout:475: read oid 6 snap -1 2023-12-15T22:58:31.327 INFO:tasks.rados.rados.0.smithi071.stdout:475: expect (ObjNum 231 snap 55 seq_num 231) 2023-12-15T22:58:31.327 INFO:tasks.rados.rados.0.smithi071.stdout:476: snap_remove snap 55 2023-12-15T22:58:31.843 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:58:31.947 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:31.945+0000 7fa1da66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:31.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:31.950+0000 7fa1da66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:32.247 INFO:tasks.rados.rados.0.smithi071.stdout:476: done (1 left) 2023-12-15T22:58:32.248 INFO:tasks.rados.rados.0.smithi071.stdout:477: read oid 25 snap 46 2023-12-15T22:58:32.248 INFO:tasks.rados.rados.0.smithi071.stdout:477: expect (ObjNum 137 snap 17 seq_num 137) 2023-12-15T22:58:32.248 INFO:tasks.rados.rados.0.smithi071.stdout:478: delete oid 2 current snap is 60 2023-12-15T22:58:32.248 INFO:tasks.rados.rados.0.smithi071.stdout:475: expect (ObjNum 231 snap 55 seq_num 231) 2023-12-15T22:58:32.501 INFO:tasks.rados.rados.0.smithi071.stdout:477: expect (ObjNum 137 snap 17 seq_num 137) 2023-12-15T22:58:32.642 INFO:tasks.rados.rados.0.smithi071.stdout:475: done (2 left) 2023-12-15T22:58:32.642 INFO:tasks.rados.rados.0.smithi071.stdout:477: done (1 left) 2023-12-15T22:58:32.642 INFO:tasks.rados.rados.0.smithi071.stdout:478: done (0 left) 2023-12-15T22:58:32.642 INFO:tasks.rados.rados.0.smithi071.stdout:479: read oid 16 snap -1 2023-12-15T22:58:32.643 INFO:tasks.rados.rados.0.smithi071.stdout:479: expect (ObjNum 176 snap 33 seq_num 176) 2023-12-15T22:58:32.643 INFO:tasks.rados.rados.0.smithi071.stdout:480: snap_create 2023-12-15T22:58:32.665 INFO:tasks.rados.rados.0.smithi071.stdout:479: expect (ObjNum 176 snap 33 seq_num 176) 2023-12-15T22:58:33.341 INFO:tasks.rados.rados.0.smithi071.stdout:479: done (1 left) 2023-12-15T22:58:33.341 INFO:tasks.rados.rados.0.smithi071.stdout:480: done (0 left) 2023-12-15T22:58:33.341 INFO:tasks.rados.rados.0.smithi071.stdout:481: snap_create 2023-12-15T22:58:34.017 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:58:34.124 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:34.123+0000 7f4d7cf5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:34.127 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:34.127+0000 7f4d7cf5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:34.341 INFO:tasks.rados.rados.0.smithi071.stdout:481: done (0 left) 2023-12-15T22:58:34.341 INFO:tasks.rados.rados.0.smithi071.stdout:482: snap_remove snap 61 2023-12-15T22:58:35.051 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T22:58:35.051 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T22:58:35.051 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-12-15T22:58:35.052 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-12-15T22:58:35.160 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:35.155+0000 7f87096f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:35.162 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:35.163+0000 7f87096f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:35.175 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:58:35.175 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:58:35.175 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:58:35.175 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:58:35.175 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:58:35.175 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:58:35.175 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:58:35.175 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:58:35.175 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:58:35.175 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:58:35.175 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:58:35.175 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:58:35.177 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:58:35.178 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:58:35.178 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-12-15T22:58:35.283 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:35.278+0000 7f6f9571e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:35.285 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:35.286+0000 7f6f9571e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:35.298 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:58:35.298 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:58:35.298 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:58:35.298 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:58:35.298 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:58:35.298 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:58:35.298 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:58:35.299 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:58:35.299 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:58:35.299 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:58:35.299 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:58:35.299 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:58:35.301 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:58:35.302 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:58:35.302 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-12-15T22:58:35.347 INFO:tasks.rados.rados.0.smithi071.stdout:482: done (0 left) 2023-12-15T22:58:35.348 INFO:tasks.rados.rados.0.smithi071.stdout:483: delete oid 48 current snap is 62 2023-12-15T22:58:35.417 INFO:tasks.rados.rados.0.smithi071.stdout:483: done (0 left) 2023-12-15T22:58:35.417 INFO:tasks.rados.rados.0.smithi071.stdout:484: read oid 46 snap -1 2023-12-15T22:58:35.417 INFO:tasks.rados.rados.0.smithi071.stdout:484: expect deleted 2023-12-15T22:58:35.417 INFO:tasks.rados.rados.0.smithi071.stdout:485: write oid 40 current snap is 62 2023-12-15T22:58:35.417 INFO:tasks.rados.rados.0.smithi071.stdout:485: seq_num 239 ranges {416873=791454,1919314=97559} 2023-12-15T22:58:35.425 INFO:tasks.rados.rados.0.smithi071.stdout:485: writing smithi07183189-40 from 416873 to 1208327 tid 1 2023-12-15T22:58:35.429 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:35.424+0000 7fee0a46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:35.430 INFO:tasks.rados.rados.0.smithi071.stdout:485: writing smithi07183189-40 from 1919314 to 2016873 tid 2 2023-12-15T22:58:35.430 INFO:tasks.rados.rados.0.smithi071.stdout:486: snap_create 2023-12-15T22:58:35.432 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:35.432+0000 7fee0a46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:35.459 INFO:tasks.rados.rados.0.smithi071.stdout:485: finishing write tid 1 to smithi07183189-40 2023-12-15T22:58:35.461 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:58:35.461 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:58:35.461 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:58:35.461 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:58:35.461 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:58:35.461 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:58:35.461 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:58:35.461 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:58:35.461 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:58:35.461 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:58:35.461 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:58:35.461 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:58:35.464 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:58:35.465 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:58:35.465 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-12-15T22:58:35.472 INFO:tasks.rados.rados.0.smithi071.stdout:485: finishing write tid 2 to smithi07183189-40 2023-12-15T22:58:35.480 INFO:tasks.rados.rados.0.smithi071.stdout:485: finishing write tid 3 to smithi07183189-40 2023-12-15T22:58:35.482 INFO:tasks.rados.rados.0.smithi071.stdout:485: finishing write tid 4 to smithi07183189-40 2023-12-15T22:58:35.482 INFO:tasks.rados.rados.0.smithi071.stdout:485: oid 40 updating version 0 to 262 2023-12-15T22:58:35.482 INFO:tasks.rados.rados.0.smithi071.stdout:485: oid 40 version 262 is already newer than 261 2023-12-15T22:58:35.482 INFO:tasks.rados.rados.0.smithi071.stdout:485: oid 40 version 262 is already newer than 260 2023-12-15T22:58:35.482 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 40 v 262 (ObjNum 239 snap 62 seq_num 239) dirty exists 2023-12-15T22:58:35.482 INFO:tasks.rados.rados.0.smithi071.stdout:485: left oid 40 (ObjNum 239 snap 62 seq_num 239) 2023-12-15T22:58:35.571 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:35.567+0000 7f7b3ba2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:35.574 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:35.574+0000 7f7b3ba2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:35.587 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:58:35.587 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:58:35.587 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:58:35.587 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:58:35.587 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:58:35.587 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:58:35.588 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:58:35.588 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:58:35.588 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:58:35.588 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:58:35.588 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:58:35.588 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:58:35.590 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:58:35.590 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:58:36.196 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:58:36.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:36.302+0000 7ffbb8f5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:36.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:36.307+0000 7ffbb8f5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:36.348 INFO:tasks.rados.rados.0.smithi071.stdout:484: done (2 left) 2023-12-15T22:58:36.348 INFO:tasks.rados.rados.0.smithi071.stdout:485: done (1 left) 2023-12-15T22:58:36.348 INFO:tasks.rados.rados.0.smithi071.stdout:486: done (0 left) 2023-12-15T22:58:36.348 INFO:tasks.rados.rados.0.smithi071.stdout:487: snap_create 2023-12-15T22:58:37.360 INFO:tasks.rados.rados.0.smithi071.stdout:487: done (0 left) 2023-12-15T22:58:37.360 INFO:tasks.rados.rados.0.smithi071.stdout:488: write (excl) oid 9 current snap is 64 2023-12-15T22:58:37.360 INFO:tasks.rados.rados.0.smithi071.stdout:488: seq_num 240 ranges {433680=466703,1577796=455884} 2023-12-15T22:58:37.364 INFO:tasks.rados.rados.0.smithi071.stdout:488: writing smithi07183189-9 from 433680 to 900383 tid 1 2023-12-15T22:58:37.371 INFO:tasks.rados.rados.0.smithi071.stdout:488: writing smithi07183189-9 from 1577796 to 2033680 tid 2 2023-12-15T22:58:37.371 INFO:tasks.rados.rados.0.smithi071.stdout:489: write oid 41 current snap is 64 2023-12-15T22:58:37.371 INFO:tasks.rados.rados.0.smithi071.stdout:489: seq_num 241 ranges {450487=541952,1636278=414209} 2023-12-15T22:58:37.377 INFO:tasks.rados.rados.0.smithi071.stdout:489: writing smithi07183189-41 from 450487 to 992439 tid 1 2023-12-15T22:58:37.385 INFO:tasks.rados.rados.0.smithi071.stdout:489: writing smithi07183189-41 from 1636278 to 2050487 tid 2 2023-12-15T22:58:37.385 INFO:tasks.rados.rados.0.smithi071.stdout:490: snap_create 2023-12-15T22:58:37.445 INFO:tasks.rados.rados.0.smithi071.stdout:488: finishing write tid 1 to smithi07183189-9 2023-12-15T22:58:37.446 INFO:tasks.rados.rados.0.smithi071.stdout:489: finishing write tid 1 to smithi07183189-41 2023-12-15T22:58:37.477 INFO:tasks.rados.rados.0.smithi071.stdout:488: finishing write tid 2 to smithi07183189-9 2023-12-15T22:58:37.477 INFO:tasks.rados.rados.0.smithi071.stdout:489: finishing write tid 2 to smithi07183189-41 2023-12-15T22:58:37.499 INFO:tasks.rados.rados.0.smithi071.stdout:488: finishing write tid 3 to smithi07183189-9 2023-12-15T22:58:37.499 INFO:tasks.rados.rados.0.smithi071.stdout:489: finishing write tid 3 to smithi07183189-41 2023-12-15T22:58:37.501 INFO:tasks.rados.rados.0.smithi071.stdout:488: finishing write tid 4 to smithi07183189-9 2023-12-15T22:58:37.501 INFO:tasks.rados.rados.0.smithi071.stdout:488: oid 9 updating version 0 to 100 2023-12-15T22:58:37.501 INFO:tasks.rados.rados.0.smithi071.stdout:488: oid 9 version 100 is already newer than 99 2023-12-15T22:58:37.501 INFO:tasks.rados.rados.0.smithi071.stdout:488: oid 9 version 100 is already newer than 98 2023-12-15T22:58:37.501 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 100 (ObjNum 240 snap 64 seq_num 240) dirty exists 2023-12-15T22:58:37.501 INFO:tasks.rados.rados.0.smithi071.stdout:488: left oid 9 (ObjNum 240 snap 64 seq_num 240) 2023-12-15T22:58:37.501 INFO:tasks.rados.rados.0.smithi071.stdout:489: finishing write tid 4 to smithi07183189-41 2023-12-15T22:58:37.501 INFO:tasks.rados.rados.0.smithi071.stdout:489: oid 41 updating version 0 to 137 2023-12-15T22:58:37.501 INFO:tasks.rados.rados.0.smithi071.stdout:489: oid 41 version 137 is already newer than 136 2023-12-15T22:58:37.501 INFO:tasks.rados.rados.0.smithi071.stdout:489: oid 41 version 137 is already newer than 135 2023-12-15T22:58:37.501 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 41 v 137 (ObjNum 241 snap 64 seq_num 241) dirty exists 2023-12-15T22:58:37.501 INFO:tasks.rados.rados.0.smithi071.stdout:489: left oid 41 (ObjNum 241 snap 64 seq_num 241) 2023-12-15T22:58:38.362 INFO:tasks.rados.rados.0.smithi071.stdout:488: done (2 left) 2023-12-15T22:58:38.362 INFO:tasks.rados.rados.0.smithi071.stdout:489: done (1 left) 2023-12-15T22:58:38.362 INFO:tasks.rados.rados.0.smithi071.stdout:490: done (0 left) 2023-12-15T22:58:38.363 INFO:tasks.rados.rados.0.smithi071.stdout:491: snap_remove snap 59 2023-12-15T22:58:38.379 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:58:38.491 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:38.490+0000 7f1ecc315640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:38.494 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:38.495+0000 7f1ecc315640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:39.368 INFO:tasks.rados.rados.0.smithi071.stdout:491: done (0 left) 2023-12-15T22:58:39.368 INFO:tasks.rados.rados.0.smithi071.stdout:492: snap_create 2023-12-15T22:58:40.369 INFO:tasks.rados.rados.0.smithi071.stdout:492: done (0 left) 2023-12-15T22:58:40.369 INFO:tasks.rados.rados.0.smithi071.stdout:493: read oid 32 snap -1 2023-12-15T22:58:40.369 INFO:tasks.rados.rados.0.smithi071.stdout:493: expect (ObjNum 174 snap 29 seq_num 174) 2023-12-15T22:58:40.369 INFO:tasks.rados.rados.0.smithi071.stdout:494: write oid 40 current snap is 66 2023-12-15T22:58:40.369 INFO:tasks.rados.rados.0.smithi071.stdout:494: seq_num 242 ranges {467294=617201,1578407=488887} 2023-12-15T22:58:40.375 INFO:tasks.rados.rados.0.smithi071.stdout:494: writing smithi07183189-40 from 467294 to 1084495 tid 1 2023-12-15T22:58:40.382 INFO:tasks.rados.rados.0.smithi071.stdout:494: writing smithi07183189-40 from 1578407 to 2067294 tid 2 2023-12-15T22:58:40.382 INFO:tasks.rados.rados.0.smithi071.stdout:495: snap_create 2023-12-15T22:58:40.421 INFO:tasks.rados.rados.0.smithi071.stdout:493: expect (ObjNum 174 snap 29 seq_num 174) 2023-12-15T22:58:40.542 INFO:tasks.rados.rados.0.smithi071.stdout:494: finishing write tid 1 to smithi07183189-40 2023-12-15T22:58:40.543 INFO:tasks.rados.rados.0.smithi071.stdout:494: finishing write tid 2 to smithi07183189-40 2023-12-15T22:58:40.543 INFO:tasks.rados.rados.0.smithi071.stdout:494: finishing write tid 3 to smithi07183189-40 2023-12-15T22:58:40.543 INFO:tasks.rados.rados.0.smithi071.stdout:494: finishing write tid 4 to smithi07183189-40 2023-12-15T22:58:40.543 INFO:tasks.rados.rados.0.smithi071.stdout:494: oid 40 updating version 0 to 268 2023-12-15T22:58:40.543 INFO:tasks.rados.rados.0.smithi071.stdout:494: oid 40 version 268 is already newer than 268 2023-12-15T22:58:40.543 INFO:tasks.rados.rados.0.smithi071.stdout:494: oid 40 version 268 is already newer than 266 2023-12-15T22:58:40.543 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 40 v 268 (ObjNum 242 snap 66 seq_num 242) dirty exists 2023-12-15T22:58:40.543 INFO:tasks.rados.rados.0.smithi071.stdout:494: left oid 40 (ObjNum 242 snap 66 seq_num 242) 2023-12-15T22:58:40.564 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:58:40.591 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T22:58:40.591 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T22:58:40.592 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T22:58:40.672 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:40.671+0000 7f291e51e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:40.675 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:40.676+0000 7f291e51e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:40.720 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:40.716+0000 7f046aaad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:40.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:40.723+0000 7f046aaad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:40.902 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:58:40.903 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-12-15T22:58:40.914 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":193,"stamp":"2023-12-15T22:58:40.537601+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":125583895,"num_objects":166,"num_object_clones":103,"num_object_copies":498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":22,"num_read":1095,"num_read_kb":720791,"num_write":961,"num_write_kb":244389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1380,"ondisk_log_size":1380,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":463148,"kb_used_data":339212,"kb_used_omap":0,"kb_used_meta":123904,"kb_avail":377024212,"statfs":{"total":386547056640,"available":386072793088,"internally_reserved":0,"allocated":347353088,"data_stored":392591161,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":126877696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3647119,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":1,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000725"},"pg_stats":[{"pgid":"3.f","version":"73'24","reported_seq":268,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-12-15T22:58:35.374481+0000","last_change":"2023-12-15T22:58:35.380287+0000","last_active":"2023-12-15T22:58:35.374481+0000","last_peered":"2023-12-15T22:58:35.374481+0000","last_clean":"2023-12-15T22:58:35.374481+0000","last_became_active":"2023-12-15T22:56:39.503797+0000","last_became_peered":"2023-12-15T22:56:39.503797+0000","last_unstale":"2023-12-15T22:58:35.374481+0000","last_undegraded":"2023-12-15T22:58:35.374481+0000","last_fullsized":"2023-12-15T22:58:35.374481+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4941225,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":38,"num_read_kb":28501,"num_write":17,"num_write_kb":4759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":231,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-12-15T22:58:35.434967+0000","last_change":"2023-12-15T22:58:35.440867+0000","last_active":"2023-12-15T22:58:35.434967+0000","last_peered":"2023-12-15T22:58:35.434967+0000","last_clean":"2023-12-15T22:58:35.434967+0000","last_became_active":"2023-12-15T22:56:39.552856+0000","last_became_peered":"2023-12-15T22:56:39.552856+0000","last_unstale":"2023-12-15T22:58:35.434967+0000","last_undegraded":"2023-12-15T22:58:35.434967+0000","last_fullsized":"2023-12-15T22:58:35.434967+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"125'174","reported_seq":419,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-12-15T22:58:39.381292+0000","last_change":"2023-12-15T22:58:39.394402+0000","last_active":"2023-12-15T22:58:39.381292+0000","last_peered":"2023-12-15T22:58:39.381292+0000","last_clean":"2023-12-15T22:58:39.381292+0000","last_became_active":"2023-12-15T22:56:39.495490+0000","last_became_peered":"2023-12-15T22:56:39.495490+0000","last_unstale":"2023-12-15T22:58:39.381292+0000","last_undegraded":"2023-12-15T22:58:39.381292+0000","last_fullsized":"2023-12-15T22:58:39.381292+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15563282,"num_objects":22,"num_object_clones":13,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":152,"num_read_kb":121013,"num_write":119,"num_write_kb":34249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"121'96","reported_seq":335,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-12-15T22:58:35.381056+0000","last_change":"2023-12-15T22:58:35.384378+0000","last_active":"2023-12-15T22:58:35.381056+0000","last_peered":"2023-12-15T22:58:35.381056+0000","last_clean":"2023-12-15T22:58:35.381056+0000","last_became_active":"2023-12-15T22:56:39.579234+0000","last_became_peered":"2023-12-15T22:56:39.579234+0000","last_unstale":"2023-12-15T22:58:35.381056+0000","last_undegraded":"2023-12-15T22:58:35.381056+0000","last_fullsized":"2023-12-15T22:58:35.381056+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9361499,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":92,"num_read_kb":73567,"num_write":66,"num_write_kb":18365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"125'71","reported_seq":287,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-15T22:58:34.357272+0000","last_change":"2023-12-15T22:58:32.648335+0000","last_active":"2023-12-15T22:58:34.357272+0000","last_peered":"2023-12-15T22:58:34.357272+0000","last_clean":"2023-12-15T22:58:34.357272+0000","last_became_active":"2023-12-15T22:56:39.572176+0000","last_became_peered":"2023-12-15T22:56:39.572176+0000","last_unstale":"2023-12-15T22:58:34.357272+0000","last_undegraded":"2023-12-15T22:58:34.357272+0000","last_fullsized":"2023-12-15T22:58:34.357272+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6319432,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":33,"num_read_kb":8584,"num_write":41,"num_write_kb":12228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"126'57","reported_seq":280,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-12-15T22:58:35.366778+0000","last_change":"2023-12-15T22:58:35.389303+0000","last_active":"2023-12-15T22:58:35.366778+0000","last_peered":"2023-12-15T22:58:35.366778+0000","last_clean":"2023-12-15T22:58:35.366778+0000","last_became_active":"2023-12-15T22:56:39.516361+0000","last_became_peered":"2023-12-15T22:56:39.516361+0000","last_unstale":"2023-12-15T22:58:35.366778+0000","last_undegraded":"2023-12-15T22:58:35.366778+0000","last_fullsized":"2023-12-15T22:58:35.366778+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3109295,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":31,"num_read_kb":25579,"num_write":35,"num_write_kb":10277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"129'262","reported_seq":444,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-12-15T22:58:35.473924+0000","last_change":"2023-12-15T22:58:35.389741+0000","last_active":"2023-12-15T22:58:35.473924+0000","last_peered":"2023-12-15T22:58:35.473924+0000","last_clean":"2023-12-15T22:58:35.473924+0000","last_became_active":"2023-12-15T22:56:39.519748+0000","last_became_peered":"2023-12-15T22:56:39.519748+0000","last_unstale":"2023-12-15T22:58:35.473924+0000","last_undegraded":"2023-12-15T22:58:35.473924+0000","last_fullsized":"2023-12-15T22:58:35.473924+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":262,"log_dups_size":0,"ondisk_log_size":262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19059171,"num_objects":34,"num_object_clones":24,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":166,"num_read_kb":105013,"num_write":148,"num_write_kb":43322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"125'59","reported_seq":314,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-12-15T22:58:39.388959+0000","last_change":"2023-12-15T22:58:39.391449+0000","last_active":"2023-12-15T22:58:39.388959+0000","last_peered":"2023-12-15T22:58:39.388959+0000","last_clean":"2023-12-15T22:58:39.388959+0000","last_became_active":"2023-12-15T22:56:39.543179+0000","last_became_peered":"2023-12-15T22:56:39.543179+0000","last_unstale":"2023-12-15T22:58:39.388959+0000","last_undegraded":"2023-12-15T22:58:39.388959+0000","last_fullsized":"2023-12-15T22:58:39.388959+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9630378,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":65511,"num_write":45,"num_write_kb":12126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":318,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-15T22:58:34.347564+0000","last_change":"2023-12-15T22:56:32.607017+0000","last_active":"2023-12-15T22:58:34.347564+0000","last_peered":"2023-12-15T22:58:34.347564+0000","last_clean":"2023-12-15T22:58:34.347564+0000","last_became_active":"2023-12-15T22:56:32.602384+0000","last_became_peered":"2023-12-15T22:56:32.602384+0000","last_unstale":"2023-12-15T22:58:34.347564+0000","last_undegraded":"2023-12-15T22:58:34.347564+0000","last_fullsized":"2023-12-15T22:58:34.347564+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_clean_scrub_stamp":"2023-12-15T22:56:31.525572+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":197,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-12-15T22:58:39.381944+0000","last_change":"2023-12-15T22:56:33.627481+0000","last_active":"2023-12-15T22:58:39.381944+0000","last_peered":"2023-12-15T22:58:39.381944+0000","last_clean":"2023-12-15T22:58:39.381944+0000","last_became_active":"2023-12-15T22:56:33.624689+0000","last_became_peered":"2023-12-15T22:56:33.624689+0000","last_unstale":"2023-12-15T22:58:39.381944+0000","last_undegraded":"2023-12-15T22:58:39.381944+0000","last_fullsized":"2023-12-15T22:58:39.381944+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"133'96","reported_seq":341,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-12-15T22:58:39.374734+0000","last_change":"2023-12-15T22:58:39.400828+0000","last_active":"2023-12-15T22:58:39.374734+0000","last_peered":"2023-12-15T22:58:39.374734+0000","last_clean":"2023-12-15T22:58:39.374734+0000","last_became_active":"2023-12-15T22:56:39.529359+0000","last_became_peered":"2023-12-15T22:56:39.529359+0000","last_unstale":"2023-12-15T22:58:39.374734+0000","last_undegraded":"2023-12-15T22:58:39.374734+0000","last_fullsized":"2023-12-15T22:58:39.374734+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9395113,"num_objects":12,"num_object_clones":6,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":70,"num_read_kb":41011,"num_write":71,"num_write_kb":21396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":191,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-12-15T22:58:35.375134+0000","last_change":"2023-12-15T22:56:33.655797+0000","last_active":"2023-12-15T22:58:35.375134+0000","last_peered":"2023-12-15T22:58:35.375134+0000","last_clean":"2023-12-15T22:58:35.375134+0000","last_became_active":"2023-12-15T22:56:33.652885+0000","last_became_peered":"2023-12-15T22:56:33.652885+0000","last_unstale":"2023-12-15T22:58:35.375134+0000","last_undegraded":"2023-12-15T22:58:35.375134+0000","last_fullsized":"2023-12-15T22:58:35.375134+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"126'133","reported_seq":345,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-15T22:58:34.357745+0000","last_change":"2023-12-15T22:58:33.643641+0000","last_active":"2023-12-15T22:58:34.357745+0000","last_peered":"2023-12-15T22:58:34.357745+0000","last_clean":"2023-12-15T22:58:34.357745+0000","last_became_active":"2023-12-15T22:56:39.543509+0000","last_became_peered":"2023-12-15T22:56:39.543509+0000","last_unstale":"2023-12-15T22:58:34.357745+0000","last_undegraded":"2023-12-15T22:58:34.357745+0000","last_fullsized":"2023-12-15T22:58:34.357745+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10722800,"num_objects":18,"num_object_clones":12,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":90,"num_read_kb":58893,"num_write":77,"num_write_kb":23428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":197,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-12-15T22:58:39.379893+0000","last_change":"2023-12-15T22:56:33.600485+0000","last_active":"2023-12-15T22:58:39.379893+0000","last_peered":"2023-12-15T22:58:39.379893+0000","last_clean":"2023-12-15T22:58:39.379893+0000","last_became_active":"2023-12-15T22:56:33.597320+0000","last_became_peered":"2023-12-15T22:56:33.597320+0000","last_unstale":"2023-12-15T22:58:39.379893+0000","last_undegraded":"2023-12-15T22:58:39.379893+0000","last_fullsized":"2023-12-15T22:58:39.379893+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"125'37","reported_seq":273,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-15T22:58:34.355649+0000","last_change":"2023-12-15T22:58:32.650528+0000","last_active":"2023-12-15T22:58:34.355649+0000","last_peered":"2023-12-15T22:58:34.355649+0000","last_clean":"2023-12-15T22:58:34.355649+0000","last_became_active":"2023-12-15T22:56:39.560980+0000","last_became_peered":"2023-12-15T22:56:39.560980+0000","last_unstale":"2023-12-15T22:58:34.355649+0000","last_undegraded":"2023-12-15T22:58:34.355649+0000","last_fullsized":"2023-12-15T22:58:34.355649+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6269011,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":37,"num_read_kb":29098,"num_write":26,"num_write_kb":7976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":191,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-12-15T22:58:35.374172+0000","last_change":"2023-12-15T22:56:33.648541+0000","last_active":"2023-12-15T22:58:35.374172+0000","last_peered":"2023-12-15T22:58:35.374172+0000","last_clean":"2023-12-15T22:58:35.374172+0000","last_became_active":"2023-12-15T22:56:33.643681+0000","last_became_peered":"2023-12-15T22:56:33.643681+0000","last_unstale":"2023-12-15T22:58:35.374172+0000","last_undegraded":"2023-12-15T22:58:35.374172+0000","last_fullsized":"2023-12-15T22:58:35.374172+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"122'85","reported_seq":316,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-12-15T22:58:35.434202+0000","last_change":"2023-12-15T22:58:35.442878+0000","last_active":"2023-12-15T22:58:35.434202+0000","last_peered":"2023-12-15T22:58:35.434202+0000","last_clean":"2023-12-15T22:58:35.434202+0000","last_became_active":"2023-12-15T22:56:39.506510+0000","last_became_peered":"2023-12-15T22:56:39.506510+0000","last_unstale":"2023-12-15T22:58:35.434202+0000","last_undegraded":"2023-12-15T22:58:35.434202+0000","last_fullsized":"2023-12-15T22:58:35.434202+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9546376,"num_objects":12,"num_object_clones":7,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":76,"num_read_kb":55523,"num_write":54,"num_write_kb":16314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"126'46","reported_seq":272,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-12-15T22:58:39.377465+0000","last_change":"2023-12-15T22:58:39.394178+0000","last_active":"2023-12-15T22:58:39.377465+0000","last_peered":"2023-12-15T22:58:39.377465+0000","last_clean":"2023-12-15T22:58:39.377465+0000","last_became_active":"2023-12-15T22:56:39.511159+0000","last_became_peered":"2023-12-15T22:56:39.511159+0000","last_unstale":"2023-12-15T22:58:39.377465+0000","last_undegraded":"2023-12-15T22:58:39.377465+0000","last_fullsized":"2023-12-15T22:58:39.377465+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3932838,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":11,"num_read_kb":6,"num_write":29,"num_write_kb":8954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":185,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-12-15T22:58:35.435682+0000","last_change":"2023-12-15T22:56:33.633890+0000","last_active":"2023-12-15T22:58:35.435682+0000","last_peered":"2023-12-15T22:58:35.435682+0000","last_clean":"2023-12-15T22:58:35.435682+0000","last_became_active":"2023-12-15T22:56:33.631134+0000","last_became_peered":"2023-12-15T22:56:33.631134+0000","last_unstale":"2023-12-15T22:58:35.435682+0000","last_undegraded":"2023-12-15T22:58:35.435682+0000","last_fullsized":"2023-12-15T22:58:35.435682+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"54'47","reported_seq":274,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-12-15T22:58:35.429633+0000","last_change":"2023-12-15T22:58:35.441394+0000","last_active":"2023-12-15T22:58:35.429633+0000","last_peered":"2023-12-15T22:58:35.429633+0000","last_clean":"2023-12-15T22:58:35.429633+0000","last_became_active":"2023-12-15T22:56:39.532730+0000","last_became_peered":"2023-12-15T22:56:39.532730+0000","last_unstale":"2023-12-15T22:58:35.429633+0000","last_undegraded":"2023-12-15T22:58:35.429633+0000","last_fullsized":"2023-12-15T22:58:35.429633+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3764735,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":41,"num_read_kb":38478,"num_write":27,"num_write_kb":8830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":189,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-15T22:58:34.351571+0000","last_change":"2023-12-15T22:56:33.655736+0000","last_active":"2023-12-15T22:58:34.351571+0000","last_peered":"2023-12-15T22:58:34.351571+0000","last_clean":"2023-12-15T22:58:34.351571+0000","last_became_active":"2023-12-15T22:56:33.652917+0000","last_became_peered":"2023-12-15T22:56:33.652917+0000","last_unstale":"2023-12-15T22:58:34.351571+0000","last_undegraded":"2023-12-15T22:58:34.351571+0000","last_fullsized":"2023-12-15T22:58:34.351571+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"122'76","reported_seq":316,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-15T22:58:34.354103+0000","last_change":"2023-12-15T22:58:32.646190+0000","last_active":"2023-12-15T22:58:34.354103+0000","last_peered":"2023-12-15T22:58:34.354103+0000","last_clean":"2023-12-15T22:58:34.354103+0000","last_became_active":"2023-12-15T22:56:39.557252+0000","last_became_peered":"2023-12-15T22:56:39.557252+0000","last_unstale":"2023-12-15T22:58:34.354103+0000","last_undegraded":"2023-12-15T22:58:34.354103+0000","last_fullsized":"2023-12-15T22:58:34.354103+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10084200,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":63,"num_read_kb":46208,"num_write":56,"num_write_kb":16343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":185,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-12-15T22:58:35.431357+0000","last_change":"2023-12-15T22:56:33.632348+0000","last_active":"2023-12-15T22:58:35.431357+0000","last_peered":"2023-12-15T22:58:35.431357+0000","last_clean":"2023-12-15T22:58:35.431357+0000","last_became_active":"2023-12-15T22:56:33.629022+0000","last_became_peered":"2023-12-15T22:56:33.629022+0000","last_unstale":"2023-12-15T22:58:35.431357+0000","last_undegraded":"2023-12-15T22:58:35.431357+0000","last_fullsized":"2023-12-15T22:58:35.431357+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"122'20","reported_seq":253,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-12-15T22:58:35.433386+0000","last_change":"2023-12-15T22:58:35.441488+0000","last_active":"2023-12-15T22:58:35.433386+0000","last_peered":"2023-12-15T22:58:35.433386+0000","last_clean":"2023-12-15T22:58:35.433386+0000","last_became_active":"2023-12-15T22:56:39.535145+0000","last_became_peered":"2023-12-15T22:56:39.535145+0000","last_unstale":"2023-12-15T22:58:35.433386+0000","last_undegraded":"2023-12-15T22:58:35.433386+0000","last_fullsized":"2023-12-15T22:58:35.433386+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3294172,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":23715,"num_write":13,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":197,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-12-15T22:58:39.379549+0000","last_change":"2023-12-15T22:56:33.627005+0000","last_active":"2023-12-15T22:58:39.379549+0000","last_peered":"2023-12-15T22:58:39.379549+0000","last_clean":"2023-12-15T22:58:39.379549+0000","last_became_active":"2023-12-15T22:56:33.622181+0000","last_became_peered":"2023-12-15T22:56:33.622181+0000","last_unstale":"2023-12-15T22:58:39.379549+0000","last_undegraded":"2023-12-15T22:58:39.379549+0000","last_fullsized":"2023-12-15T22:58:39.379549+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":124993527,"num_objects":164,"num_object_clones":103,"num_object_copies":492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":22,"num_read":992,"num_read_kb":720700,"num_write":824,"num_write_kb":242495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1283,"ondisk_log_size":1283,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705789,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":122456,"kb_used_data":91472,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94249384,"statfs":{"total":96636764160,"available":96511369216,"internally_reserved":0,"allocated":93667328,"data_stored":105581560,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705793,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":95944,"kb_used_data":64960,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94275896,"statfs":{"total":96636764160,"available":96538517504,"internally_reserved":0,"allocated":66519040,"data_stored":75435538,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705793,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":124372,"kb_used_data":93388,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94247468,"statfs":{"total":96636764160,"available":96509407232,"internally_reserved":0,"allocated":95629312,"data_stored":110694726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705789,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":120376,"kb_used_data":89392,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94251464,"statfs":{"total":96636764160,"available":96513499136,"internally_reserved":0,"allocated":91537408,"data_stored":100879337,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-15T22:58:41.383 INFO:tasks.rados.rados.0.smithi071.stdout:493: done (2 left) 2023-12-15T22:58:41.383 INFO:tasks.rados.rados.0.smithi071.stdout:494: done (1 left) 2023-12-15T22:58:41.383 INFO:tasks.rados.rados.0.smithi071.stdout:495: done (0 left) 2023-12-15T22:58:41.383 INFO:tasks.rados.rados.0.smithi071.stdout:496: read oid 38 snap -1 2023-12-15T22:58:41.383 INFO:tasks.rados.rados.0.smithi071.stdout:496: expect (ObjNum 54 snap 0 seq_num 54) 2023-12-15T22:58:41.383 INFO:tasks.rados.rados.0.smithi071.stdout:497: snap_remove snap 56 2023-12-15T22:58:42.251 INFO:tasks.rados.rados.0.smithi071.stdout:497: done (1 left) 2023-12-15T22:58:42.251 INFO:tasks.rados.rados.0.smithi071.stdout:498: write oid 9 current snap is 67 2023-12-15T22:58:42.251 INFO:tasks.rados.rados.0.smithi071.stdout:498: seq_num 243 ranges {484101=692450,1636889=447212} 2023-12-15T22:58:42.257 INFO:tasks.rados.rados.0.smithi071.stdout:498: writing smithi07183189-9 from 484101 to 1176551 tid 1 2023-12-15T22:58:42.263 INFO:tasks.rados.rados.0.smithi071.stdout:498: writing smithi07183189-9 from 1636889 to 2084101 tid 2 2023-12-15T22:58:42.263 INFO:tasks.rados.rados.0.smithi071.stdout:499: write oid 34 current snap is 67 2023-12-15T22:58:42.263 INFO:tasks.rados.rados.0.smithi071.stdout:499: seq_num 244 ranges {500908=484052,1411724=689184} 2023-12-15T22:58:42.268 INFO:tasks.rados.rados.0.smithi071.stdout:499: writing smithi07183189-34 from 500908 to 984960 tid 1 2023-12-15T22:58:42.275 INFO:tasks.rados.rados.0.smithi071.stdout:499: writing smithi07183189-34 from 1411724 to 2100908 tid 2 2023-12-15T22:58:42.275 INFO:tasks.rados.rados.0.smithi071.stdout:500: snap_create 2023-12-15T22:58:42.275 INFO:tasks.rados.rados.0.smithi071.stdout:496: expect (ObjNum 54 snap 0 seq_num 54) 2023-12-15T22:58:42.467 INFO:tasks.rados.rados.0.smithi071.stdout:499: finishing write tid 1 to smithi07183189-34 2023-12-15T22:58:42.467 INFO:tasks.rados.rados.0.smithi071.stdout:498: finishing write tid 1 to smithi07183189-9 2023-12-15T22:58:42.468 INFO:tasks.rados.rados.0.smithi071.stdout:499: finishing write tid 2 to smithi07183189-34 2023-12-15T22:58:42.468 INFO:tasks.rados.rados.0.smithi071.stdout:498: finishing write tid 2 to smithi07183189-9 2023-12-15T22:58:42.468 INFO:tasks.rados.rados.0.smithi071.stdout:499: finishing write tid 3 to smithi07183189-34 2023-12-15T22:58:42.468 INFO:tasks.rados.rados.0.smithi071.stdout:499: finishing write tid 4 to smithi07183189-34 2023-12-15T22:58:42.468 INFO:tasks.rados.rados.0.smithi071.stdout:499: oid 34 updating version 0 to 23 2023-12-15T22:58:42.468 INFO:tasks.rados.rados.0.smithi071.stdout:499: oid 34 version 23 is already newer than 22 2023-12-15T22:58:42.468 INFO:tasks.rados.rados.0.smithi071.stdout:499: oid 34 updating version 23 to 24 2023-12-15T22:58:42.468 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 24 (ObjNum 244 snap 67 seq_num 244) dirty exists 2023-12-15T22:58:42.468 INFO:tasks.rados.rados.0.smithi071.stdout:499: left oid 34 (ObjNum 244 snap 67 seq_num 244) 2023-12-15T22:58:42.468 INFO:tasks.rados.rados.0.smithi071.stdout:498: finishing write tid 3 to smithi07183189-9 2023-12-15T22:58:42.468 INFO:tasks.rados.rados.0.smithi071.stdout:498: finishing write tid 4 to smithi07183189-9 2023-12-15T22:58:42.468 INFO:tasks.rados.rados.0.smithi071.stdout:498: oid 9 updating version 0 to 105 2023-12-15T22:58:42.468 INFO:tasks.rados.rados.0.smithi071.stdout:498: oid 9 updating version 105 to 106 2023-12-15T22:58:42.468 INFO:tasks.rados.rados.0.smithi071.stdout:498: oid 9 version 106 is already newer than 105 2023-12-15T22:58:42.468 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 106 (ObjNum 243 snap 67 seq_num 243) dirty exists 2023-12-15T22:58:42.469 INFO:tasks.rados.rados.0.smithi071.stdout:498: left oid 9 (ObjNum 243 snap 67 seq_num 243) 2023-12-15T22:58:42.754 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:58:42.858 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:42.857+0000 7f6ce419e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:42.861 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:42.861+0000 7f6ce419e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:43.255 INFO:tasks.rados.rados.0.smithi071.stdout:496: done (3 left) 2023-12-15T22:58:43.255 INFO:tasks.rados.rados.0.smithi071.stdout:498: done (2 left) 2023-12-15T22:58:43.255 INFO:tasks.rados.rados.0.smithi071.stdout:499: done (1 left) 2023-12-15T22:58:43.255 INFO:tasks.rados.rados.0.smithi071.stdout:500: done (0 left) 2023-12-15T22:58:43.255 INFO:tasks.rados.rados.0.smithi071.stdout:501: write (excl) oid 44 current snap is 68 2023-12-15T22:58:43.255 INFO:tasks.rados.rados.0.smithi071.stdout:501: seq_num 245 ranges {517715=559301,1870206=247509} 2023-12-15T22:58:43.261 INFO:tasks.rados.rados.0.smithi071.stdout:501: writing smithi07183189-44 from 517715 to 1077016 tid 1 2023-12-15T22:58:43.267 INFO:tasks.rados.rados.0.smithi071.stdout:501: writing smithi07183189-44 from 1870206 to 2117715 tid 2 2023-12-15T22:58:43.267 INFO:tasks.rados.rados.0.smithi071.stdout:502: write oid 49 current snap is 68 2023-12-15T22:58:43.267 INFO:tasks.rados.rados.0.smithi071.stdout:502: seq_num 246 ranges {534522=634550,1812335=322187} 2023-12-15T22:58:43.273 INFO:tasks.rados.rados.0.smithi071.stdout:502: writing smithi07183189-49 from 534522 to 1169072 tid 1 2023-12-15T22:58:43.279 INFO:tasks.rados.rados.0.smithi071.stdout:502: writing smithi07183189-49 from 1812335 to 2134522 tid 2 2023-12-15T22:58:43.279 INFO:tasks.rados.rados.0.smithi071.stdout:503: read oid 17 snap -1 2023-12-15T22:58:43.279 INFO:tasks.rados.rados.0.smithi071.stdout:503: expect (ObjNum 220 snap 49 seq_num 220) 2023-12-15T22:58:43.279 INFO:tasks.rados.rados.0.smithi071.stdout:504: read oid 34 snap -1 2023-12-15T22:58:43.279 INFO:tasks.rados.rados.0.smithi071.stdout:504: expect (ObjNum 244 snap 67 seq_num 244) 2023-12-15T22:58:43.279 INFO:tasks.rados.rados.0.smithi071.stdout:505: read oid 21 snap -1 2023-12-15T22:58:43.280 INFO:tasks.rados.rados.0.smithi071.stdout:505: expect (ObjNum 55 snap 0 seq_num 55) 2023-12-15T22:58:43.280 INFO:tasks.rados.rados.0.smithi071.stdout:506: delete oid 30 current snap is 68 2023-12-15T22:58:43.302 INFO:tasks.rados.rados.0.smithi071.stdout:503: expect (ObjNum 220 snap 49 seq_num 220) 2023-12-15T22:58:43.589 INFO:tasks.rados.rados.0.smithi071.stdout:505: expect (ObjNum 55 snap 0 seq_num 55) 2023-12-15T22:58:43.735 INFO:tasks.rados.rados.0.smithi071.stdout:501: finishing write tid 1 to smithi07183189-44 2023-12-15T22:58:43.735 INFO:tasks.rados.rados.0.smithi071.stdout:504: expect (ObjNum 244 snap 67 seq_num 244) 2023-12-15T22:58:43.842 INFO:tasks.rados.rados.0.smithi071.stdout:502: finishing write tid 1 to smithi07183189-49 2023-12-15T22:58:43.842 INFO:tasks.rados.rados.0.smithi071.stdout:501: finishing write tid 2 to smithi07183189-44 2023-12-15T22:58:43.842 INFO:tasks.rados.rados.0.smithi071.stdout:501: finishing write tid 3 to smithi07183189-44 2023-12-15T22:58:43.842 INFO:tasks.rados.rados.0.smithi071.stdout:501: finishing write tid 4 to smithi07183189-44 2023-12-15T22:58:43.843 INFO:tasks.rados.rados.0.smithi071.stdout:501: oid 44 updating version 0 to 51 2023-12-15T22:58:43.843 INFO:tasks.rados.rados.0.smithi071.stdout:501: oid 44 version 51 is already newer than 49 2023-12-15T22:58:43.843 INFO:tasks.rados.rados.0.smithi071.stdout:501: oid 44 version 51 is already newer than 51 2023-12-15T22:58:43.843 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 51 (ObjNum 245 snap 68 seq_num 245) dirty exists 2023-12-15T22:58:43.843 INFO:tasks.rados.rados.0.smithi071.stdout:501: left oid 44 (ObjNum 245 snap 68 seq_num 245) 2023-12-15T22:58:43.843 INFO:tasks.rados.rados.0.smithi071.stdout:502: finishing write tid 2 to smithi07183189-49 2023-12-15T22:58:43.843 INFO:tasks.rados.rados.0.smithi071.stdout:502: finishing write tid 3 to smithi07183189-49 2023-12-15T22:58:43.843 INFO:tasks.rados.rados.0.smithi071.stdout:501: done (5 left) 2023-12-15T22:58:43.843 INFO:tasks.rados.rados.0.smithi071.stdout:503: done (4 left) 2023-12-15T22:58:43.843 INFO:tasks.rados.rados.0.smithi071.stdout:504: done (3 left) 2023-12-15T22:58:43.843 INFO:tasks.rados.rados.0.smithi071.stdout:505: done (2 left) 2023-12-15T22:58:43.843 INFO:tasks.rados.rados.0.smithi071.stdout:506: done (1 left) 2023-12-15T22:58:43.843 INFO:tasks.rados.rados.0.smithi071.stdout:507: write (excl) oid 2 current snap is 68 2023-12-15T22:58:43.843 INFO:tasks.rados.rados.0.smithi071.stdout:502: finishing write tid 4 to smithi07183189-49 2023-12-15T22:58:43.843 INFO:tasks.rados.rados.0.smithi071.stdout:502: oid 49 updating version 0 to 89 2023-12-15T22:58:43.844 INFO:tasks.rados.rados.0.smithi071.stdout:502: oid 49 version 89 is already newer than 87 2023-12-15T22:58:43.844 INFO:tasks.rados.rados.0.smithi071.stdout:502: oid 49 version 89 is already newer than 88 2023-12-15T22:58:43.844 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 49 v 89 (ObjNum 246 snap 68 seq_num 246) dirty exists 2023-12-15T22:58:43.844 INFO:tasks.rados.rados.0.smithi071.stdout:502: left oid 49 (ObjNum 246 snap 68 seq_num 246) 2023-12-15T22:58:43.844 INFO:tasks.rados.rados.0.smithi071.stdout:507: seq_num 248 ranges {568136=785048,1929299=238837} 2023-12-15T22:58:43.849 INFO:tasks.rados.rados.0.smithi071.stdout:507: writing smithi07183189-2 from 568136 to 1353184 tid 1 2023-12-15T22:58:43.853 INFO:tasks.rados.rados.0.smithi071.stdout:507: writing smithi07183189-2 from 1929299 to 2168136 tid 2 2023-12-15T22:58:43.853 INFO:tasks.rados.rados.0.smithi071.stdout:502: done (1 left) 2023-12-15T22:58:43.854 INFO:tasks.rados.rados.0.smithi071.stdout:508: write (excl) oid 30 current snap is 68 2023-12-15T22:58:43.854 INFO:tasks.rados.rados.0.smithi071.stdout:508: seq_num 249 ranges {584943=460297,1587781=597162} 2023-12-15T22:58:43.859 INFO:tasks.rados.rados.0.smithi071.stdout:508: writing smithi07183189-30 from 584943 to 1045240 tid 1 2023-12-15T22:58:43.867 INFO:tasks.rados.rados.0.smithi071.stdout:508: writing smithi07183189-30 from 1587781 to 2184943 tid 2 2023-12-15T22:58:43.867 INFO:tasks.rados.rados.0.smithi071.stdout:509: snap_create 2023-12-15T22:58:43.875 INFO:tasks.rados.rados.0.smithi071.stdout:508: finishing write tid 1 to smithi07183189-30 2023-12-15T22:58:43.888 INFO:tasks.rados.rados.0.smithi071.stdout:508: finishing write tid 2 to smithi07183189-30 2023-12-15T22:58:43.899 INFO:tasks.rados.rados.0.smithi071.stdout:508: finishing write tid 3 to smithi07183189-30 2023-12-15T22:58:43.900 INFO:tasks.rados.rados.0.smithi071.stdout:508: finishing write tid 4 to smithi07183189-30 2023-12-15T22:58:43.900 INFO:tasks.rados.rados.0.smithi071.stdout:508: oid 30 updating version 0 to 41 2023-12-15T22:58:43.900 INFO:tasks.rados.rados.0.smithi071.stdout:508: oid 30 version 41 is already newer than 40 2023-12-15T22:58:43.901 INFO:tasks.rados.rados.0.smithi071.stdout:508: oid 30 updating version 41 to 42 2023-12-15T22:58:43.901 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 42 (ObjNum 249 snap 68 seq_num 249) dirty exists 2023-12-15T22:58:43.901 INFO:tasks.rados.rados.0.smithi071.stdout:508: left oid 30 (ObjNum 249 snap 68 seq_num 249) 2023-12-15T22:58:43.906 INFO:tasks.rados.rados.0.smithi071.stdout:507: finishing write tid 1 to smithi07183189-2 2023-12-15T22:58:43.917 INFO:tasks.rados.rados.0.smithi071.stdout:507: finishing write tid 2 to smithi07183189-2 2023-12-15T22:58:43.923 INFO:tasks.rados.rados.0.smithi071.stdout:507: finishing write tid 3 to smithi07183189-2 2023-12-15T22:58:43.924 INFO:tasks.rados.rados.0.smithi071.stdout:507: finishing write tid 4 to smithi07183189-2 2023-12-15T22:58:43.924 INFO:tasks.rados.rados.0.smithi071.stdout:507: oid 2 updating version 0 to 59 2023-12-15T22:58:43.924 INFO:tasks.rados.rados.0.smithi071.stdout:507: oid 2 updating version 59 to 60 2023-12-15T22:58:43.924 INFO:tasks.rados.rados.0.smithi071.stdout:507: oid 2 version 60 is already newer than 58 2023-12-15T22:58:43.924 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 60 (ObjNum 248 snap 68 seq_num 248) dirty exists 2023-12-15T22:58:43.925 INFO:tasks.rados.rados.0.smithi071.stdout:507: left oid 2 (ObjNum 248 snap 68 seq_num 248) 2023-12-15T22:58:44.387 INFO:tasks.rados.rados.0.smithi071.stdout:507: done (2 left) 2023-12-15T22:58:44.387 INFO:tasks.rados.rados.0.smithi071.stdout:508: done (1 left) 2023-12-15T22:58:44.387 INFO:tasks.rados.rados.0.smithi071.stdout:509: done (0 left) 2023-12-15T22:58:44.387 INFO:tasks.rados.rados.0.smithi071.stdout:510: write oid 18 current snap is 69 2023-12-15T22:58:44.388 INFO:tasks.rados.rados.0.smithi071.stdout:510: seq_num 250 ranges {601750=535546,1929910=271840} 2023-12-15T22:58:44.393 INFO:tasks.rados.rados.0.smithi071.stdout:510: writing smithi07183189-18 from 601750 to 1137296 tid 1 2023-12-15T22:58:44.399 INFO:tasks.rados.rados.0.smithi071.stdout:510: writing smithi07183189-18 from 1929910 to 2201750 tid 2 2023-12-15T22:58:44.400 INFO:tasks.rados.rados.0.smithi071.stdout:511: read oid 47 snap -1 2023-12-15T22:58:44.400 INFO:tasks.rados.rados.0.smithi071.stdout:511: expect (ObjNum 193 snap 38 seq_num 193) 2023-12-15T22:58:44.400 INFO:tasks.rados.rados.0.smithi071.stdout:512: snap_remove snap 62 2023-12-15T22:58:44.928 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:58:45.032 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:45.031+0000 7fabdb264640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:45.036 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:45.035+0000 7fabdb264640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:45.396 INFO:tasks.rados.rados.0.smithi071.stdout:512: done (2 left) 2023-12-15T22:58:45.396 INFO:tasks.rados.rados.0.smithi071.stdout:513: read oid 41 snap -1 2023-12-15T22:58:45.396 INFO:tasks.rados.rados.0.smithi071.stdout:513: expect (ObjNum 241 snap 64 seq_num 241) 2023-12-15T22:58:45.396 INFO:tasks.rados.rados.0.smithi071.stdout:514: delete oid 31 current snap is 69 2023-12-15T22:58:45.396 INFO:tasks.rados.rados.0.smithi071.stdout:511: expect (ObjNum 193 snap 38 seq_num 193) 2023-12-15T22:58:45.564 INFO:tasks.rados.rados.0.smithi071.stdout:510: finishing write tid 1 to smithi07183189-18 2023-12-15T22:58:45.564 INFO:tasks.rados.rados.0.smithi071.stdout:510: finishing write tid 2 to smithi07183189-18 2023-12-15T22:58:45.564 INFO:tasks.rados.rados.0.smithi071.stdout:510: finishing write tid 3 to smithi07183189-18 2023-12-15T22:58:45.564 INFO:tasks.rados.rados.0.smithi071.stdout:510: finishing write tid 4 to smithi07183189-18 2023-12-15T22:58:45.564 INFO:tasks.rados.rados.0.smithi071.stdout:510: oid 18 updating version 0 to 63 2023-12-15T22:58:45.564 INFO:tasks.rados.rados.0.smithi071.stdout:510: oid 18 version 63 is already newer than 62 2023-12-15T22:58:45.565 INFO:tasks.rados.rados.0.smithi071.stdout:510: oid 18 updating version 63 to 64 2023-12-15T22:58:45.565 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 64 (ObjNum 250 snap 69 seq_num 250) dirty exists 2023-12-15T22:58:45.565 INFO:tasks.rados.rados.0.smithi071.stdout:510: left oid 18 (ObjNum 250 snap 69 seq_num 250) 2023-12-15T22:58:45.565 INFO:tasks.rados.rados.0.smithi071.stdout:510: done (3 left) 2023-12-15T22:58:45.565 INFO:tasks.rados.rados.0.smithi071.stdout:511: done (2 left) 2023-12-15T22:58:45.565 INFO:tasks.rados.rados.0.smithi071.stdout:514: done (1 left) 2023-12-15T22:58:45.565 INFO:tasks.rados.rados.0.smithi071.stdout:515: write oid 40 current snap is 69 2023-12-15T22:58:45.565 INFO:tasks.rados.rados.0.smithi071.stdout:515: seq_num 252 ranges {635364=402397,1763227=472137} 2023-12-15T22:58:45.569 INFO:tasks.rados.rados.0.smithi071.stdout:515: writing smithi07183189-40 from 635364 to 1037761 tid 1 2023-12-15T22:58:45.577 INFO:tasks.rados.rados.0.smithi071.stdout:515: writing smithi07183189-40 from 1763227 to 2235364 tid 2 2023-12-15T22:58:45.577 INFO:tasks.rados.rados.0.smithi071.stdout:516: snap_create 2023-12-15T22:58:45.577 INFO:tasks.rados.rados.0.smithi071.stdout:513: expect (ObjNum 241 snap 64 seq_num 241) 2023-12-15T22:58:45.705 INFO:tasks.rados.rados.0.smithi071.stdout:515: finishing write tid 1 to smithi07183189-40 2023-12-15T22:58:45.705 INFO:tasks.rados.rados.0.smithi071.stdout:515: finishing write tid 2 to smithi07183189-40 2023-12-15T22:58:45.705 INFO:tasks.rados.rados.0.smithi071.stdout:515: finishing write tid 3 to smithi07183189-40 2023-12-15T22:58:45.705 INFO:tasks.rados.rados.0.smithi071.stdout:515: finishing write tid 4 to smithi07183189-40 2023-12-15T22:58:45.705 INFO:tasks.rados.rados.0.smithi071.stdout:515: oid 40 updating version 0 to 279 2023-12-15T22:58:45.705 INFO:tasks.rados.rados.0.smithi071.stdout:515: oid 40 version 279 is already newer than 278 2023-12-15T22:58:45.705 INFO:tasks.rados.rados.0.smithi071.stdout:515: oid 40 updating version 279 to 280 2023-12-15T22:58:45.705 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 40 v 280 (ObjNum 252 snap 69 seq_num 252) dirty exists 2023-12-15T22:58:45.705 INFO:tasks.rados.rados.0.smithi071.stdout:515: left oid 40 (ObjNum 252 snap 69 seq_num 252) 2023-12-15T22:58:45.916 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T22:58:45.917 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T22:58:45.917 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T22:58:46.021 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:46.018+0000 7f04fe66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:46.024 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:46.024+0000 7f04fe66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:46.198 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:58:46.198 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-12-15T22:58:46.210 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2023-12-15T22:58:45.392374+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":119852939,"num_objects":172,"num_object_clones":109,"num_object_copies":516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":22,"num_read":1127,"num_read_kb":749250,"num_write":985,"num_write_kb":250178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1414,"ondisk_log_size":1414,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":475156,"kb_used_data":351220,"kb_used_omap":0,"kb_used_meta":123904,"kb_avail":377012204,"statfs":{"total":386547056640,"available":386060496896,"internally_reserved":0,"allocated":359649280,"data_stored":409951518,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":126877696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3109163,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":19888,"num_write":12,"num_write_kb":2871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.010698"},"pg_stats":[{"pgid":"3.f","version":"73'24","reported_seq":278,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-15T22:58:40.416310+0000","last_change":"2023-12-15T22:58:39.536444+0000","last_active":"2023-12-15T22:58:40.416310+0000","last_peered":"2023-12-15T22:58:40.416310+0000","last_clean":"2023-12-15T22:58:40.416310+0000","last_became_active":"2023-12-15T22:56:39.503797+0000","last_became_peered":"2023-12-15T22:56:39.503797+0000","last_unstale":"2023-12-15T22:58:40.416310+0000","last_undegraded":"2023-12-15T22:58:40.416310+0000","last_fullsized":"2023-12-15T22:58:40.416310+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4941225,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":44,"num_read_kb":37069,"num_write":17,"num_write_kb":4759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":243,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-12-15T22:58:43.280186+0000","last_change":"2023-12-15T22:58:42.301474+0000","last_active":"2023-12-15T22:58:43.280186+0000","last_peered":"2023-12-15T22:58:43.280186+0000","last_clean":"2023-12-15T22:58:43.280186+0000","last_became_active":"2023-12-15T22:56:39.552856+0000","last_became_peered":"2023-12-15T22:56:39.552856+0000","last_unstale":"2023-12-15T22:58:43.280186+0000","last_undegraded":"2023-12-15T22:58:43.280186+0000","last_fullsized":"2023-12-15T22:58:43.280186+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"125'174","reported_seq":429,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-12-15T22:58:43.298635+0000","last_change":"2023-12-15T22:58:43.277514+0000","last_active":"2023-12-15T22:58:43.298635+0000","last_peered":"2023-12-15T22:58:43.298635+0000","last_clean":"2023-12-15T22:58:43.298635+0000","last_became_active":"2023-12-15T22:56:39.495490+0000","last_became_peered":"2023-12-15T22:56:39.495490+0000","last_unstale":"2023-12-15T22:58:43.298635+0000","last_undegraded":"2023-12-15T22:58:43.298635+0000","last_fullsized":"2023-12-15T22:58:43.298635+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15563282,"num_objects":22,"num_object_clones":13,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":158,"num_read_kb":129032,"num_write":119,"num_write_kb":34249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"133'102","reported_seq":346,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-15T22:58:40.385205+0000","last_change":"2023-12-15T22:58:39.636383+0000","last_active":"2023-12-15T22:58:40.385205+0000","last_peered":"2023-12-15T22:58:40.385205+0000","last_clean":"2023-12-15T22:58:40.385205+0000","last_became_active":"2023-12-15T22:56:39.579234+0000","last_became_peered":"2023-12-15T22:56:39.579234+0000","last_unstale":"2023-12-15T22:58:40.385205+0000","last_undegraded":"2023-12-15T22:58:40.385205+0000","last_fullsized":"2023-12-15T22:58:40.385205+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8537989,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":94,"num_read_kb":73568,"num_write":70,"num_write_kb":19267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"125'71","reported_seq":295,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-15T22:58:40.377834+0000","last_change":"2023-12-15T22:58:39.539798+0000","last_active":"2023-12-15T22:58:40.377834+0000","last_peered":"2023-12-15T22:58:40.377834+0000","last_clean":"2023-12-15T22:58:40.377834+0000","last_became_active":"2023-12-15T22:56:39.572176+0000","last_became_peered":"2023-12-15T22:56:39.572176+0000","last_unstale":"2023-12-15T22:58:40.377834+0000","last_undegraded":"2023-12-15T22:58:40.377834+0000","last_fullsized":"2023-12-15T22:58:40.377834+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6319432,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":33,"num_read_kb":8584,"num_write":41,"num_write_kb":12228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"126'57","reported_seq":287,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-15T22:58:40.380555+0000","last_change":"2023-12-15T22:58:39.537805+0000","last_active":"2023-12-15T22:58:40.380555+0000","last_peered":"2023-12-15T22:58:40.380555+0000","last_clean":"2023-12-15T22:58:40.380555+0000","last_became_active":"2023-12-15T22:56:39.516361+0000","last_became_peered":"2023-12-15T22:56:39.516361+0000","last_unstale":"2023-12-15T22:58:40.380555+0000","last_undegraded":"2023-12-15T22:58:40.380555+0000","last_fullsized":"2023-12-15T22:58:40.380555+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3109295,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":31,"num_read_kb":25579,"num_write":35,"num_write_kb":10277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"134'268","reported_seq":456,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-15T22:58:40.441745+0000","last_change":"2023-12-15T22:58:39.625246+0000","last_active":"2023-12-15T22:58:40.441745+0000","last_peered":"2023-12-15T22:58:40.441745+0000","last_clean":"2023-12-15T22:58:40.441745+0000","last_became_active":"2023-12-15T22:56:39.519748+0000","last_became_peered":"2023-12-15T22:56:39.519748+0000","last_unstale":"2023-12-15T22:58:40.441745+0000","last_undegraded":"2023-12-15T22:58:40.441745+0000","last_fullsized":"2023-12-15T22:58:40.441745+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19109592,"num_objects":35,"num_object_clones":25,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":167,"num_read_kb":105014,"num_write":152,"num_write_kb":44403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"125'59","reported_seq":324,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-12-15T22:58:43.276628+0000","last_change":"2023-12-15T22:58:43.279196+0000","last_active":"2023-12-15T22:58:43.276628+0000","last_peered":"2023-12-15T22:58:43.276628+0000","last_clean":"2023-12-15T22:58:43.276628+0000","last_became_active":"2023-12-15T22:56:39.543179+0000","last_became_peered":"2023-12-15T22:56:39.543179+0000","last_unstale":"2023-12-15T22:58:43.276628+0000","last_undegraded":"2023-12-15T22:58:43.276628+0000","last_fullsized":"2023-12-15T22:58:43.276628+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9630378,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":76,"num_read_kb":71703,"num_write":45,"num_write_kb":12126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":324,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-15T22:58:40.381957+0000","last_change":"2023-12-15T22:56:32.607017+0000","last_active":"2023-12-15T22:58:40.381957+0000","last_peered":"2023-12-15T22:58:40.381957+0000","last_clean":"2023-12-15T22:58:40.381957+0000","last_became_active":"2023-12-15T22:56:32.602384+0000","last_became_peered":"2023-12-15T22:56:32.602384+0000","last_unstale":"2023-12-15T22:58:40.381957+0000","last_undegraded":"2023-12-15T22:58:40.381957+0000","last_fullsized":"2023-12-15T22:58:40.381957+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_clean_scrub_stamp":"2023-12-15T22:56:31.525572+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":203,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-12-15T22:58:43.274240+0000","last_change":"2023-12-15T22:56:33.627481+0000","last_active":"2023-12-15T22:58:43.274240+0000","last_peered":"2023-12-15T22:58:43.274240+0000","last_clean":"2023-12-15T22:58:43.274240+0000","last_became_active":"2023-12-15T22:56:33.624689+0000","last_became_peered":"2023-12-15T22:56:33.624689+0000","last_unstale":"2023-12-15T22:58:43.274240+0000","last_undegraded":"2023-12-15T22:58:43.274240+0000","last_fullsized":"2023-12-15T22:58:43.274240+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"137'98","reported_seq":348,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-12-15T22:58:43.269350+0000","last_change":"2023-12-15T22:58:43.289172+0000","last_active":"2023-12-15T22:58:43.269350+0000","last_peered":"2023-12-15T22:58:43.269350+0000","last_clean":"2023-12-15T22:58:43.269350+0000","last_became_active":"2023-12-15T22:56:39.529359+0000","last_became_peered":"2023-12-15T22:56:39.529359+0000","last_unstale":"2023-12-15T22:58:43.269350+0000","last_undegraded":"2023-12-15T22:58:43.269350+0000","last_fullsized":"2023-12-15T22:58:43.269350+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9395113,"num_objects":12,"num_object_clones":6,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":70,"num_read_kb":41011,"num_write":71,"num_write_kb":21396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":197,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-15T22:58:40.388777+0000","last_change":"2023-12-15T22:56:33.655797+0000","last_active":"2023-12-15T22:58:40.388777+0000","last_peered":"2023-12-15T22:58:40.388777+0000","last_clean":"2023-12-15T22:58:40.388777+0000","last_became_active":"2023-12-15T22:56:33.652885+0000","last_became_peered":"2023-12-15T22:56:33.652885+0000","last_unstale":"2023-12-15T22:58:40.388777+0000","last_undegraded":"2023-12-15T22:58:40.388777+0000","last_fullsized":"2023-12-15T22:58:40.388777+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"133'139","reported_seq":357,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-15T22:58:40.374986+0000","last_change":"2023-12-15T22:58:39.601252+0000","last_active":"2023-12-15T22:58:40.374986+0000","last_peered":"2023-12-15T22:58:40.374986+0000","last_clean":"2023-12-15T22:58:40.374986+0000","last_became_active":"2023-12-15T22:56:39.543509+0000","last_became_peered":"2023-12-15T22:56:39.543509+0000","last_unstale":"2023-12-15T22:58:40.374986+0000","last_undegraded":"2023-12-15T22:58:40.374986+0000","last_fullsized":"2023-12-15T22:58:40.374986+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8874096,"num_objects":19,"num_object_clones":13,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":91,"num_read_kb":58894,"num_write":81,"num_write_kb":24363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":203,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-12-15T22:58:43.272802+0000","last_change":"2023-12-15T22:56:33.600485+0000","last_active":"2023-12-15T22:58:43.272802+0000","last_peered":"2023-12-15T22:58:43.272802+0000","last_clean":"2023-12-15T22:58:43.272802+0000","last_became_active":"2023-12-15T22:56:33.597320+0000","last_became_peered":"2023-12-15T22:56:33.597320+0000","last_unstale":"2023-12-15T22:58:43.272802+0000","last_undegraded":"2023-12-15T22:58:43.272802+0000","last_fullsized":"2023-12-15T22:58:43.272802+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"125'37","reported_seq":281,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-15T22:58:40.384188+0000","last_change":"2023-12-15T22:58:39.540249+0000","last_active":"2023-12-15T22:58:40.384188+0000","last_peered":"2023-12-15T22:58:40.384188+0000","last_clean":"2023-12-15T22:58:40.384188+0000","last_became_active":"2023-12-15T22:56:39.560980+0000","last_became_peered":"2023-12-15T22:56:39.560980+0000","last_unstale":"2023-12-15T22:58:40.384188+0000","last_undegraded":"2023-12-15T22:58:40.384188+0000","last_fullsized":"2023-12-15T22:58:40.384188+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6269011,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":37,"num_read_kb":29098,"num_write":26,"num_write_kb":7976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":197,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-15T22:58:40.382086+0000","last_change":"2023-12-15T22:56:33.648541+0000","last_active":"2023-12-15T22:58:40.382086+0000","last_peered":"2023-12-15T22:58:40.382086+0000","last_clean":"2023-12-15T22:58:40.382086+0000","last_became_active":"2023-12-15T22:56:33.643681+0000","last_became_peered":"2023-12-15T22:56:33.643681+0000","last_unstale":"2023-12-15T22:58:40.382086+0000","last_undegraded":"2023-12-15T22:58:40.382086+0000","last_fullsized":"2023-12-15T22:58:40.382086+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"137'89","reported_seq":332,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-12-15T22:58:43.415968+0000","last_change":"2023-12-15T22:58:42.306631+0000","last_active":"2023-12-15T22:58:43.415968+0000","last_peered":"2023-12-15T22:58:43.415968+0000","last_clean":"2023-12-15T22:58:43.415968+0000","last_became_active":"2023-12-15T22:56:39.506510+0000","last_became_peered":"2023-12-15T22:56:39.506510+0000","last_unstale":"2023-12-15T22:58:43.415968+0000","last_undegraded":"2023-12-15T22:58:43.415968+0000","last_fullsized":"2023-12-15T22:58:43.415968+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9277497,"num_objects":13,"num_object_clones":8,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":77,"num_read_kb":55524,"num_write":58,"num_write_kb":17249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"137'48","reported_seq":279,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-12-15T22:58:43.270212+0000","last_change":"2023-12-15T22:58:43.316080+0000","last_active":"2023-12-15T22:58:43.270212+0000","last_peered":"2023-12-15T22:58:43.270212+0000","last_clean":"2023-12-15T22:58:43.270212+0000","last_became_active":"2023-12-15T22:56:39.511159+0000","last_became_peered":"2023-12-15T22:56:39.511159+0000","last_unstale":"2023-12-15T22:58:43.270212+0000","last_undegraded":"2023-12-15T22:58:43.270212+0000","last_fullsized":"2023-12-15T22:58:43.270212+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3932838,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":11,"num_read_kb":6,"num_write":29,"num_write_kb":8954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":195,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-12-15T22:58:43.281005+0000","last_change":"2023-12-15T22:56:33.633890+0000","last_active":"2023-12-15T22:58:43.281005+0000","last_peered":"2023-12-15T22:58:43.281005+0000","last_clean":"2023-12-15T22:58:43.281005+0000","last_became_active":"2023-12-15T22:56:33.631134+0000","last_became_peered":"2023-12-15T22:56:33.631134+0000","last_unstale":"2023-12-15T22:58:43.281005+0000","last_undegraded":"2023-12-15T22:58:43.281005+0000","last_fullsized":"2023-12-15T22:58:43.281005+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"137'51","reported_seq":290,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-12-15T22:58:43.384378+0000","last_change":"2023-12-15T22:58:42.302277+0000","last_active":"2023-12-15T22:58:43.384378+0000","last_peered":"2023-12-15T22:58:43.384378+0000","last_clean":"2023-12-15T22:58:43.384378+0000","last_became_active":"2023-12-15T22:56:39.532730+0000","last_became_peered":"2023-12-15T22:56:39.532730+0000","last_unstale":"2023-12-15T22:58:43.384378+0000","last_undegraded":"2023-12-15T22:58:43.384378+0000","last_fullsized":"2023-12-15T22:58:43.384378+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2117715,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":43,"num_read_kb":38479,"num_write":31,"num_write_kb":9619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":195,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-15T22:58:40.386301+0000","last_change":"2023-12-15T22:56:33.655736+0000","last_active":"2023-12-15T22:58:40.386301+0000","last_peered":"2023-12-15T22:58:40.386301+0000","last_clean":"2023-12-15T22:58:40.386301+0000","last_became_active":"2023-12-15T22:56:33.652917+0000","last_became_peered":"2023-12-15T22:56:33.652917+0000","last_unstale":"2023-12-15T22:58:40.386301+0000","last_undegraded":"2023-12-15T22:58:40.386301+0000","last_fullsized":"2023-12-15T22:58:40.386301+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"122'76","reported_seq":324,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-15T22:58:40.383228+0000","last_change":"2023-12-15T22:58:39.540895+0000","last_active":"2023-12-15T22:58:40.383228+0000","last_peered":"2023-12-15T22:58:40.383228+0000","last_clean":"2023-12-15T22:58:40.383228+0000","last_became_active":"2023-12-15T22:56:39.557252+0000","last_became_peered":"2023-12-15T22:56:39.557252+0000","last_unstale":"2023-12-15T22:58:40.383228+0000","last_undegraded":"2023-12-15T22:58:40.383228+0000","last_fullsized":"2023-12-15T22:58:40.383228+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10084200,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":63,"num_read_kb":46208,"num_write":56,"num_write_kb":16343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":195,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-12-15T22:58:43.278187+0000","last_change":"2023-12-15T22:56:33.632348+0000","last_active":"2023-12-15T22:58:43.278187+0000","last_peered":"2023-12-15T22:58:43.278187+0000","last_clean":"2023-12-15T22:58:43.278187+0000","last_became_active":"2023-12-15T22:56:33.629022+0000","last_became_peered":"2023-12-15T22:56:33.629022+0000","last_unstale":"2023-12-15T22:58:43.278187+0000","last_undegraded":"2023-12-15T22:58:43.278187+0000","last_fullsized":"2023-12-15T22:58:43.278187+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"136'24","reported_seq":272,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-12-15T22:58:43.384026+0000","last_change":"2023-12-15T22:58:42.301636+0000","last_active":"2023-12-15T22:58:43.384026+0000","last_peered":"2023-12-15T22:58:43.384026+0000","last_clean":"2023-12-15T22:58:43.384026+0000","last_became_active":"2023-12-15T22:56:39.535145+0000","last_became_peered":"2023-12-15T22:56:39.535145+0000","last_unstale":"2023-12-15T22:58:43.384026+0000","last_undegraded":"2023-12-15T22:58:43.384026+0000","last_fullsized":"2023-12-15T22:58:43.384026+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2100908,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":29,"num_read_kb":29390,"num_write":17,"num_write_kb":5075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":203,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-12-15T22:58:43.265712+0000","last_change":"2023-12-15T22:56:33.627005+0000","last_active":"2023-12-15T22:58:43.265712+0000","last_peered":"2023-12-15T22:58:43.265712+0000","last_clean":"2023-12-15T22:58:43.265712+0000","last_became_active":"2023-12-15T22:56:33.622181+0000","last_became_peered":"2023-12-15T22:56:33.622181+0000","last_unstale":"2023-12-15T22:58:43.265712+0000","last_undegraded":"2023-12-15T22:58:43.265712+0000","last_fullsized":"2023-12-15T22:58:43.265712+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":119262571,"num_objects":170,"num_object_clones":109,"num_object_copies":510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":22,"num_read":1024,"num_read_kb":749159,"num_write":848,"num_write_kb":248284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1317,"ondisk_log_size":1317,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705794,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":124328,"kb_used_data":93344,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94247512,"statfs":{"total":96636764160,"available":96509452288,"internally_reserved":0,"allocated":95584256,"data_stored":108444648,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705798,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":101072,"kb_used_data":70088,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94270768,"statfs":{"total":96636764160,"available":96533266432,"internally_reserved":0,"allocated":71770112,"data_stored":82790151,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705798,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":126628,"kb_used_data":95644,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94245212,"statfs":{"total":96636764160,"available":96507097088,"internally_reserved":0,"allocated":97939456,"data_stored":113671829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705794,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123128,"kb_used_data":92144,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94248712,"statfs":{"total":96636764160,"available":96510681088,"internally_reserved":0,"allocated":94355456,"data_stored":105044890,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-15T22:58:46.401 INFO:tasks.rados.rados.0.smithi071.stdout:513: done (2 left) 2023-12-15T22:58:46.401 INFO:tasks.rados.rados.0.smithi071.stdout:515: done (1 left) 2023-12-15T22:58:46.401 INFO:tasks.rados.rados.0.smithi071.stdout:516: done (0 left) 2023-12-15T22:58:46.401 INFO:tasks.rados.rados.0.smithi071.stdout:517: snap_remove snap 60 2023-12-15T22:58:47.105 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:58:47.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:47.209+0000 7f8c43108640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:47.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:47.214+0000 7f8c43108640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:47.407 INFO:tasks.rados.rados.0.smithi071.stdout:517: done (0 left) 2023-12-15T22:58:47.407 INFO:tasks.rados.rados.0.smithi071.stdout:518: snap_remove snap 39 2023-12-15T22:58:48.409 INFO:tasks.rados.rados.0.smithi071.stdout:518: done (0 left) 2023-12-15T22:58:48.409 INFO:tasks.rados.rados.0.smithi071.stdout:519: write (excl) oid 22 current snap is 70 2023-12-15T22:58:48.409 INFO:tasks.rados.rados.0.smithi071.stdout:519: seq_num 253 ranges {652171=477646,1821709=430462} 2023-12-15T22:58:48.416 INFO:tasks.rados.rados.0.smithi071.stdout:519: writing smithi07183189-22 from 652171 to 1129817 tid 1 2023-12-15T22:58:48.424 INFO:tasks.rados.rados.0.smithi071.stdout:519: writing smithi07183189-22 from 1821709 to 2252171 tid 2 2023-12-15T22:58:48.424 INFO:tasks.rados.rados.0.smithi071.stdout:520: read oid 23 snap 50 2023-12-15T22:58:48.424 INFO:tasks.rados.rados.0.smithi071.stdout:520: expect (ObjNum 206 snap 44 seq_num 206) 2023-12-15T22:58:48.424 INFO:tasks.rados.rados.0.smithi071.stdout:521: read oid 16 snap -1 2023-12-15T22:58:48.424 INFO:tasks.rados.rados.0.smithi071.stdout:521: expect (ObjNum 176 snap 33 seq_num 176) 2023-12-15T22:58:48.424 INFO:tasks.rados.rados.0.smithi071.stdout:522: read oid 1 snap -1 2023-12-15T22:58:48.424 INFO:tasks.rados.rados.0.smithi071.stdout:522: expect (ObjNum 0 snap 0 seq_num 0) 2023-12-15T22:58:48.424 INFO:tasks.rados.rados.0.smithi071.stdout:523: delete oid 45 current snap is 70 2023-12-15T22:58:48.485 INFO:tasks.rados.rados.0.smithi071.stdout:522: expect (ObjNum 0 snap 0 seq_num 0) 2023-12-15T22:58:48.572 INFO:tasks.rados.rados.0.smithi071.stdout:520: expect (ObjNum 206 snap 44 seq_num 206) 2023-12-15T22:58:48.720 INFO:tasks.rados.rados.0.smithi071.stdout:521: expect (ObjNum 176 snap 33 seq_num 176) 2023-12-15T22:58:48.837 INFO:tasks.rados.rados.0.smithi071.stdout:519: finishing write tid 1 to smithi07183189-22 2023-12-15T22:58:48.837 INFO:tasks.rados.rados.0.smithi071.stdout:519: finishing write tid 2 to smithi07183189-22 2023-12-15T22:58:48.837 INFO:tasks.rados.rados.0.smithi071.stdout:519: finishing write tid 3 to smithi07183189-22 2023-12-15T22:58:48.837 INFO:tasks.rados.rados.0.smithi071.stdout:519: finishing write tid 4 to smithi07183189-22 2023-12-15T22:58:48.837 INFO:tasks.rados.rados.0.smithi071.stdout:519: oid 22 updating version 0 to 98 2023-12-15T22:58:48.837 INFO:tasks.rados.rados.0.smithi071.stdout:519: oid 22 version 98 is already newer than 97 2023-12-15T22:58:48.837 INFO:tasks.rados.rados.0.smithi071.stdout:519: oid 22 updating version 98 to 99 2023-12-15T22:58:48.838 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 99 (ObjNum 253 snap 70 seq_num 253) dirty exists 2023-12-15T22:58:48.838 INFO:tasks.rados.rados.0.smithi071.stdout:519: left oid 22 (ObjNum 253 snap 70 seq_num 253) 2023-12-15T22:58:48.838 INFO:tasks.rados.rados.0.smithi071.stdout:519: done (4 left) 2023-12-15T22:58:48.838 INFO:tasks.rados.rados.0.smithi071.stdout:520: done (3 left) 2023-12-15T22:58:48.838 INFO:tasks.rados.rados.0.smithi071.stdout:521: done (2 left) 2023-12-15T22:58:48.838 INFO:tasks.rados.rados.0.smithi071.stdout:522: done (1 left) 2023-12-15T22:58:48.838 INFO:tasks.rados.rados.0.smithi071.stdout:523: done (0 left) 2023-12-15T22:58:48.838 INFO:tasks.rados.rados.0.smithi071.stdout:524: write (excl) oid 18 current snap is 70 2023-12-15T22:58:48.838 INFO:tasks.rados.rados.0.smithi071.stdout:524: seq_num 255 ranges {685785=628144,1822320=463465} 2023-12-15T22:58:48.843 INFO:tasks.rados.rados.0.smithi071.stdout:524: writing smithi07183189-18 from 685785 to 1313929 tid 1 2023-12-15T22:58:48.849 INFO:tasks.rados.rados.0.smithi071.stdout:524: writing smithi07183189-18 from 1822320 to 2285785 tid 2 2023-12-15T22:58:48.849 INFO:tasks.rados.rados.0.smithi071.stdout:525: snap_remove snap 65 2023-12-15T22:58:49.283 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:58:49.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:49.385+0000 7fcf4539e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:49.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:49.390+0000 7fcf4539e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:49.424 INFO:tasks.rados.rados.0.smithi071.stdout:525: done (1 left) 2023-12-15T22:58:49.424 INFO:tasks.rados.rados.0.smithi071.stdout:526: read oid 41 snap 66 2023-12-15T22:58:49.424 INFO:tasks.rados.rados.0.smithi071.stdout:526: expect (ObjNum 241 snap 64 seq_num 241) 2023-12-15T22:58:49.424 INFO:tasks.rados.rados.0.smithi071.stdout:527: read oid 15 snap -1 2023-12-15T22:58:49.425 INFO:tasks.rados.rados.0.smithi071.stdout:527: expect (ObjNum 197 snap 40 seq_num 197) 2023-12-15T22:58:49.425 INFO:tasks.rados.rados.0.smithi071.stdout:528: read oid 7 snap 46 2023-12-15T22:58:49.425 INFO:tasks.rados.rados.0.smithi071.stdout:528: expect (ObjNum 167 snap 26 seq_num 167) 2023-12-15T22:58:49.425 INFO:tasks.rados.rados.0.smithi071.stdout:529: write oid 26 current snap is 70 2023-12-15T22:58:49.425 INFO:tasks.rados.rados.0.smithi071.stdout:524: finishing write tid 1 to smithi07183189-18 2023-12-15T22:58:49.425 INFO:tasks.rados.rados.0.smithi071.stdout:529: seq_num 256 ranges {702592=703393,1880802=421790} 2023-12-15T22:58:49.432 INFO:tasks.rados.rados.0.smithi071.stdout:529: writing smithi07183189-26 from 702592 to 1405985 tid 1 2023-12-15T22:58:49.438 INFO:tasks.rados.rados.0.smithi071.stdout:529: writing smithi07183189-26 from 1880802 to 2302592 tid 2 2023-12-15T22:58:49.439 INFO:tasks.rados.rados.0.smithi071.stdout:530: snap_remove snap 64 2023-12-15T22:58:50.424 INFO:tasks.rados.rados.0.smithi071.stdout:530: done (5 left) 2023-12-15T22:58:50.424 INFO:tasks.rados.rados.0.smithi071.stdout:531: write oid 9 current snap is 70 2023-12-15T22:58:50.424 INFO:tasks.rados.rados.0.smithi071.stdout:524: finishing write tid 2 to smithi07183189-18 2023-12-15T22:58:50.424 INFO:tasks.rados.rados.0.smithi071.stdout:531: seq_num 257 ranges {719399=778642,1939284=380115} 2023-12-15T22:58:50.434 INFO:tasks.rados.rados.0.smithi071.stdout:531: writing smithi07183189-9 from 719399 to 1498041 tid 1 2023-12-15T22:58:50.439 INFO:tasks.rados.rados.0.smithi071.stdout:531: writing smithi07183189-9 from 1939284 to 2319399 tid 2 2023-12-15T22:58:50.439 INFO:tasks.rados.rados.0.smithi071.stdout:532: read oid 48 snap 54 2023-12-15T22:58:50.439 INFO:tasks.rados.rados.0.smithi071.stdout:532: expect (ObjNum 217 snap 47 seq_num 217) 2023-12-15T22:58:50.440 INFO:tasks.rados.rados.0.smithi071.stdout:533: delete oid 47 current snap is 70 2023-12-15T22:58:50.440 INFO:tasks.rados.rados.0.smithi071.stdout:524: finishing write tid 3 to smithi07183189-18 2023-12-15T22:58:50.440 INFO:tasks.rados.rados.0.smithi071.stdout:524: finishing write tid 4 to smithi07183189-18 2023-12-15T22:58:50.440 INFO:tasks.rados.rados.0.smithi071.stdout:524: oid 18 updating version 0 to 74 2023-12-15T22:58:50.440 INFO:tasks.rados.rados.0.smithi071.stdout:524: oid 18 version 74 is already newer than 73 2023-12-15T22:58:50.440 INFO:tasks.rados.rados.0.smithi071.stdout:524: oid 18 version 74 is already newer than 72 2023-12-15T22:58:50.440 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 74 (ObjNum 255 snap 70 seq_num 255) dirty exists 2023-12-15T22:58:50.440 INFO:tasks.rados.rados.0.smithi071.stdout:524: left oid 18 (ObjNum 255 snap 70 seq_num 255) 2023-12-15T22:58:50.440 INFO:tasks.rados.rados.0.smithi071.stdout:527: expect (ObjNum 197 snap 40 seq_num 197) 2023-12-15T22:58:50.583 INFO:tasks.rados.rados.0.smithi071.stdout:528: expect (ObjNum 167 snap 26 seq_num 167) 2023-12-15T22:58:50.739 INFO:tasks.rados.rados.0.smithi071.stdout:526: expect (ObjNum 241 snap 64 seq_num 241) 2023-12-15T22:58:50.857 INFO:tasks.rados.rados.0.smithi071.stdout:529: finishing write tid 1 to smithi07183189-26 2023-12-15T22:58:50.858 INFO:tasks.rados.rados.0.smithi071.stdout:529: finishing write tid 2 to smithi07183189-26 2023-12-15T22:58:50.858 INFO:tasks.rados.rados.0.smithi071.stdout:529: finishing write tid 3 to smithi07183189-26 2023-12-15T22:58:50.858 INFO:tasks.rados.rados.0.smithi071.stdout:529: finishing write tid 4 to smithi07183189-26 2023-12-15T22:58:50.858 INFO:tasks.rados.rados.0.smithi071.stdout:529: oid 26 updating version 0 to 87 2023-12-15T22:58:50.858 INFO:tasks.rados.rados.0.smithi071.stdout:529: oid 26 updating version 87 to 88 2023-12-15T22:58:50.858 INFO:tasks.rados.rados.0.smithi071.stdout:529: oid 26 version 88 is already newer than 86 2023-12-15T22:58:50.858 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 88 (ObjNum 256 snap 70 seq_num 256) dirty exists 2023-12-15T22:58:50.858 INFO:tasks.rados.rados.0.smithi071.stdout:529: left oid 26 (ObjNum 256 snap 70 seq_num 256) 2023-12-15T22:58:50.858 INFO:tasks.rados.rados.0.smithi071.stdout:524: done (7 left) 2023-12-15T22:58:50.858 INFO:tasks.rados.rados.0.smithi071.stdout:526: done (6 left) 2023-12-15T22:58:50.858 INFO:tasks.rados.rados.0.smithi071.stdout:527: done (5 left) 2023-12-15T22:58:50.858 INFO:tasks.rados.rados.0.smithi071.stdout:528: done (4 left) 2023-12-15T22:58:50.858 INFO:tasks.rados.rados.0.smithi071.stdout:529: done (3 left) 2023-12-15T22:58:50.859 INFO:tasks.rados.rados.0.smithi071.stdout:533: done (2 left) 2023-12-15T22:58:50.859 INFO:tasks.rados.rados.0.smithi071.stdout:534: write (excl) oid 26 current snap is 70 2023-12-15T22:58:50.859 INFO:tasks.rados.rados.0.smithi071.stdout:532: expect (ObjNum 217 snap 47 seq_num 217) 2023-12-15T22:58:51.051 INFO:tasks.rados.rados.0.smithi071.stdout:531: finishing write tid 1 to smithi07183189-9 2023-12-15T22:58:51.051 INFO:tasks.rados.rados.0.smithi071.stdout:531: finishing write tid 2 to smithi07183189-9 2023-12-15T22:58:51.051 INFO:tasks.rados.rados.0.smithi071.stdout:531: finishing write tid 3 to smithi07183189-9 2023-12-15T22:58:51.051 INFO:tasks.rados.rados.0.smithi071.stdout:531: finishing write tid 4 to smithi07183189-9 2023-12-15T22:58:51.052 INFO:tasks.rados.rados.0.smithi071.stdout:531: oid 9 updating version 0 to 122 2023-12-15T22:58:51.052 INFO:tasks.rados.rados.0.smithi071.stdout:531: oid 9 version 122 is already newer than 120 2023-12-15T22:58:51.052 INFO:tasks.rados.rados.0.smithi071.stdout:531: oid 9 version 122 is already newer than 121 2023-12-15T22:58:51.052 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 122 (ObjNum 257 snap 70 seq_num 257) dirty exists 2023-12-15T22:58:51.052 INFO:tasks.rados.rados.0.smithi071.stdout:531: left oid 9 (ObjNum 257 snap 70 seq_num 257) 2023-12-15T22:58:51.052 INFO:tasks.rados.rados.0.smithi071.stdout:534: seq_num 259 ranges {753013=645493,2056248=296765} 2023-12-15T22:58:51.059 INFO:tasks.rados.rados.0.smithi071.stdout:534: writing smithi07183189-26 from 753013 to 1398506 tid 1 2023-12-15T22:58:51.064 INFO:tasks.rados.rados.0.smithi071.stdout:534: writing smithi07183189-26 from 2056248 to 2353013 tid 2 2023-12-15T22:58:51.064 INFO:tasks.rados.rados.0.smithi071.stdout:531: done (2 left) 2023-12-15T22:58:51.064 INFO:tasks.rados.rados.0.smithi071.stdout:532: done (1 left) 2023-12-15T22:58:51.065 INFO:tasks.rados.rados.0.smithi071.stdout:535: write (excl) oid 6 current snap is 70 2023-12-15T22:58:51.065 INFO:tasks.rados.rados.0.smithi071.stdout:535: seq_num 260 ranges {769820=720742,2114730=255090} 2023-12-15T22:58:51.072 INFO:tasks.rados.rados.0.smithi071.stdout:535: writing smithi07183189-6 from 769820 to 1490562 tid 1 2023-12-15T22:58:51.077 INFO:tasks.rados.rados.0.smithi071.stdout:535: writing smithi07183189-6 from 2114730 to 2369820 tid 2 2023-12-15T22:58:51.077 INFO:tasks.rados.rados.0.smithi071.stdout:536: delete oid 31 current snap is 70 2023-12-15T22:58:51.107 INFO:tasks.rados.rados.0.smithi071.stdout:534: finishing write tid 1 to smithi07183189-26 2023-12-15T22:58:51.109 INFO:tasks.rados.rados.0.smithi071.stdout:535: finishing write tid 1 to smithi07183189-6 2023-12-15T22:58:51.128 INFO:tasks.rados.rados.0.smithi071.stdout:534: finishing write tid 2 to smithi07183189-26 2023-12-15T22:58:51.134 INFO:tasks.rados.rados.0.smithi071.stdout:535: finishing write tid 2 to smithi07183189-6 2023-12-15T22:58:51.134 INFO:tasks.rados.rados.0.smithi071.stdout:534: finishing write tid 3 to smithi07183189-26 2023-12-15T22:58:51.136 INFO:tasks.rados.rados.0.smithi071.stdout:534: finishing write tid 4 to smithi07183189-26 2023-12-15T22:58:51.136 INFO:tasks.rados.rados.0.smithi071.stdout:534: oid 26 updating version 0 to 93 2023-12-15T22:58:51.137 INFO:tasks.rados.rados.0.smithi071.stdout:534: oid 26 updating version 93 to 95 2023-12-15T22:58:51.137 INFO:tasks.rados.rados.0.smithi071.stdout:534: oid 26 version 95 is already newer than 94 2023-12-15T22:58:51.137 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 95 (ObjNum 259 snap 70 seq_num 259) dirty exists 2023-12-15T22:58:51.137 INFO:tasks.rados.rados.0.smithi071.stdout:534: left oid 26 (ObjNum 259 snap 70 seq_num 259) 2023-12-15T22:58:51.137 INFO:tasks.rados.rados.0.smithi071.stdout:534: done (2 left) 2023-12-15T22:58:51.137 INFO:tasks.rados.rados.0.smithi071.stdout:536: done (1 left) 2023-12-15T22:58:51.137 INFO:tasks.rados.rados.0.smithi071.stdout:537: read oid 10 snap -1 2023-12-15T22:58:51.137 INFO:tasks.rados.rados.0.smithi071.stdout:537: expect (ObjNum 147 snap 20 seq_num 147) 2023-12-15T22:58:51.137 INFO:tasks.rados.rados.0.smithi071.stdout:538: write oid 24 current snap is 70 2023-12-15T22:58:51.137 INFO:tasks.rados.rados.0.smithi071.stdout:538: seq_num 262 ranges {403434=471240,1315341=400756} 2023-12-15T22:58:51.141 INFO:tasks.rados.rados.0.smithi071.stdout:538: writing smithi07183189-24 from 403434 to 874674 tid 1 2023-12-15T22:58:51.146 INFO:tasks.rados.rados.0.smithi071.stdout:538: writing smithi07183189-24 from 1315341 to 1716097 tid 2 2023-12-15T22:58:51.146 INFO:tasks.rados.rados.0.smithi071.stdout:539: snap_remove snap 54 2023-12-15T22:58:51.212 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T22:58:51.212 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T22:58:51.213 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T22:58:51.318 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:51.314+0000 7f1c3d8de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:51.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:51.321+0000 7f1c3d8de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:51.430 INFO:tasks.rados.rados.0.smithi071.stdout:539: done (3 left) 2023-12-15T22:58:51.430 INFO:tasks.rados.rados.0.smithi071.stdout:540: delete oid 40 current snap is 70 2023-12-15T22:58:51.430 INFO:tasks.rados.rados.0.smithi071.stdout:537: expect (ObjNum 147 snap 20 seq_num 147) 2023-12-15T22:58:51.512 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:58:51.512 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-12-15T22:58:51.525 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":209,"stamp":"2023-12-15T22:58:51.428216+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":108810905,"num_objects":184,"num_object_clones":120,"num_object_copies":552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":25,"num_read":1197,"num_read_kb":805127,"num_write":1021,"num_write_kb":259158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1580,"ondisk_log_size":1580,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":489308,"kb_used_data":365372,"kb_used_omap":0,"kb_used_meta":123904,"kb_avail":376998052,"statfs":{"total":386547056640,"available":386046005248,"internally_reserved":0,"allocated":374140928,"data_stored":426693900,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":126877696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8437048,"num_objects":7,"num_object_clones":7,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":44,"num_read_kb":36427,"num_write":20,"num_write_kb":5042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.022612"},"pg_stats":[{"pgid":"3.f","version":"73'24","reported_seq":307,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-12-15T22:58:50.461821+0000","last_change":"2023-12-15T22:58:50.474141+0000","last_active":"2023-12-15T22:58:50.461821+0000","last_peered":"2023-12-15T22:58:50.461821+0000","last_clean":"2023-12-15T22:58:50.461821+0000","last_became_active":"2023-12-15T22:56:39.503797+0000","last_became_peered":"2023-12-15T22:56:39.503797+0000","last_unstale":"2023-12-15T22:58:50.461821+0000","last_undegraded":"2023-12-15T22:58:50.461821+0000","last_fullsized":"2023-12-15T22:58:50.461821+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4941225,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":50,"num_read_kb":39853,"num_write":17,"num_write_kb":4759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":257,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-15T22:58:49.459741+0000","last_change":"2023-12-15T22:58:49.468015+0000","last_active":"2023-12-15T22:58:49.459741+0000","last_peered":"2023-12-15T22:58:49.459741+0000","last_clean":"2023-12-15T22:58:49.459741+0000","last_became_active":"2023-12-15T22:56:39.552856+0000","last_became_peered":"2023-12-15T22:56:39.552856+0000","last_unstale":"2023-12-15T22:58:49.459741+0000","last_undegraded":"2023-12-15T22:58:49.459741+0000","last_fullsized":"2023-12-15T22:58:49.459741+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"142'180","reported_seq":447,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-15T22:58:49.460321+0000","last_change":"2023-12-15T22:58:49.443272+0000","last_active":"2023-12-15T22:58:49.460321+0000","last_peered":"2023-12-15T22:58:49.460321+0000","last_clean":"2023-12-15T22:58:49.460321+0000","last_became_active":"2023-12-15T22:56:39.495490+0000","last_became_peered":"2023-12-15T22:56:39.495490+0000","last_unstale":"2023-12-15T22:58:49.460321+0000","last_undegraded":"2023-12-15T22:58:49.460321+0000","last_fullsized":"2023-12-15T22:58:49.460321+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15563282,"num_objects":22,"num_object_clones":13,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":170,"num_read_kb":141220,"num_write":119,"num_write_kb":34249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"144'124","reported_seq":380,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-12-15T22:58:50.564523+0000","last_change":"2023-12-15T22:58:50.566700+0000","last_active":"2023-12-15T22:58:50.564523+0000","last_peered":"2023-12-15T22:58:50.564523+0000","last_clean":"2023-12-15T22:58:50.564523+0000","last_became_active":"2023-12-15T22:56:39.579234+0000","last_became_peered":"2023-12-15T22:56:39.579234+0000","last_unstale":"2023-12-15T22:58:50.564523+0000","last_undegraded":"2023-12-15T22:58:50.564523+0000","last_fullsized":"2023-12-15T22:58:50.564523+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8823708,"num_objects":14,"num_object_clones":11,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":96,"num_read_kb":73570,"num_write":78,"num_write_kb":21514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"125'71","reported_seq":325,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-12-15T22:58:50.463231+0000","last_change":"2023-12-15T22:58:50.463231+0000","last_active":"2023-12-15T22:58:50.463231+0000","last_peered":"2023-12-15T22:58:50.463231+0000","last_clean":"2023-12-15T22:58:50.463231+0000","last_became_active":"2023-12-15T22:56:39.572176+0000","last_became_peered":"2023-12-15T22:56:39.572176+0000","last_unstale":"2023-12-15T22:58:50.463231+0000","last_undegraded":"2023-12-15T22:58:50.463231+0000","last_fullsized":"2023-12-15T22:58:50.463231+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3075681,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":39,"num_read_kb":16821,"num_write":41,"num_write_kb":12228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"144'78","reported_seq":325,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-12-15T22:58:50.463649+0000","last_change":"2023-12-15T22:58:50.526681+0000","last_active":"2023-12-15T22:58:50.463649+0000","last_peered":"2023-12-15T22:58:50.463649+0000","last_clean":"2023-12-15T22:58:50.463649+0000","last_became_active":"2023-12-15T22:56:39.516361+0000","last_became_peered":"2023-12-15T22:56:39.516361+0000","last_unstale":"2023-12-15T22:58:50.463649+0000","last_undegraded":"2023-12-15T22:58:50.463649+0000","last_fullsized":"2023-12-15T22:58:50.463649+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4453921,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":36,"num_read_kb":25582,"num_write":47,"num_write_kb":13134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"144'302","reported_seq":487,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-12-15T22:58:50.449924+0000","last_change":"2023-12-15T22:58:50.529445+0000","last_active":"2023-12-15T22:58:50.449924+0000","last_peered":"2023-12-15T22:58:50.449924+0000","last_clean":"2023-12-15T22:58:50.449924+0000","last_became_active":"2023-12-15T22:56:39.519748+0000","last_became_peered":"2023-12-15T22:56:39.519748+0000","last_unstale":"2023-12-15T22:58:50.449924+0000","last_undegraded":"2023-12-15T22:58:50.449924+0000","last_fullsized":"2023-12-15T22:58:50.449924+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":302,"log_dups_size":0,"ondisk_log_size":302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15361631,"num_objects":37,"num_object_clones":27,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":169,"num_read_kb":105015,"num_write":156,"num_write_kb":45258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"142'61","reported_seq":336,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-15T22:58:49.443204+0000","last_change":"2023-12-15T22:58:49.446901+0000","last_active":"2023-12-15T22:58:49.443204+0000","last_peered":"2023-12-15T22:58:49.443204+0000","last_clean":"2023-12-15T22:58:49.443204+0000","last_became_active":"2023-12-15T22:56:39.543179+0000","last_became_peered":"2023-12-15T22:56:39.543179+0000","last_unstale":"2023-12-15T22:58:49.443204+0000","last_undegraded":"2023-12-15T22:58:49.443204+0000","last_fullsized":"2023-12-15T22:58:49.443204+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9630378,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":76,"num_read_kb":71703,"num_write":45,"num_write_kb":12126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":344,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-12-15T22:58:50.462252+0000","last_change":"2023-12-15T22:56:32.607017+0000","last_active":"2023-12-15T22:58:50.462252+0000","last_peered":"2023-12-15T22:58:50.462252+0000","last_clean":"2023-12-15T22:58:50.462252+0000","last_became_active":"2023-12-15T22:56:32.602384+0000","last_became_peered":"2023-12-15T22:56:32.602384+0000","last_unstale":"2023-12-15T22:58:50.462252+0000","last_undegraded":"2023-12-15T22:58:50.462252+0000","last_fullsized":"2023-12-15T22:58:50.462252+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_clean_scrub_stamp":"2023-12-15T22:56:31.525572+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":211,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-15T22:58:49.444553+0000","last_change":"2023-12-15T22:56:33.627481+0000","last_active":"2023-12-15T22:58:49.444553+0000","last_peered":"2023-12-15T22:58:49.444553+0000","last_clean":"2023-12-15T22:58:49.444553+0000","last_became_active":"2023-12-15T22:56:33.624689+0000","last_became_peered":"2023-12-15T22:56:33.624689+0000","last_unstale":"2023-12-15T22:58:49.444553+0000","last_undegraded":"2023-12-15T22:58:49.444553+0000","last_fullsized":"2023-12-15T22:58:49.444553+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"142'102","reported_seq":363,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-15T22:58:49.453771+0000","last_change":"2023-12-15T22:58:49.440123+0000","last_active":"2023-12-15T22:58:49.453771+0000","last_peered":"2023-12-15T22:58:49.453771+0000","last_clean":"2023-12-15T22:58:49.453771+0000","last_became_active":"2023-12-15T22:56:39.529359+0000","last_became_peered":"2023-12-15T22:56:39.529359+0000","last_unstale":"2023-12-15T22:58:49.453771+0000","last_undegraded":"2023-12-15T22:58:49.453771+0000","last_fullsized":"2023-12-15T22:58:49.453771+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9395113,"num_objects":12,"num_object_clones":6,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":76,"num_read_kb":48998,"num_write":71,"num_write_kb":21396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":217,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-12-15T22:58:50.466433+0000","last_change":"2023-12-15T22:56:33.655797+0000","last_active":"2023-12-15T22:58:50.466433+0000","last_peered":"2023-12-15T22:58:50.466433+0000","last_clean":"2023-12-15T22:58:50.466433+0000","last_became_active":"2023-12-15T22:56:33.652885+0000","last_became_peered":"2023-12-15T22:56:33.652885+0000","last_unstale":"2023-12-15T22:58:50.466433+0000","last_undegraded":"2023-12-15T22:58:50.466433+0000","last_fullsized":"2023-12-15T22:58:50.466433+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"142'151","reported_seq":392,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-12-15T22:58:50.458079+0000","last_change":"2023-12-15T22:58:50.467170+0000","last_active":"2023-12-15T22:58:50.458079+0000","last_peered":"2023-12-15T22:58:50.458079+0000","last_clean":"2023-12-15T22:58:50.458079+0000","last_became_active":"2023-12-15T22:56:39.543509+0000","last_became_peered":"2023-12-15T22:56:39.543509+0000","last_unstale":"2023-12-15T22:58:50.458079+0000","last_undegraded":"2023-12-15T22:58:50.458079+0000","last_fullsized":"2023-12-15T22:58:50.458079+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8874096,"num_objects":19,"num_object_clones":13,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":109,"num_read_kb":75504,"num_write":81,"num_write_kb":24363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":211,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-15T22:58:49.439377+0000","last_change":"2023-12-15T22:56:33.600485+0000","last_active":"2023-12-15T22:58:49.439377+0000","last_peered":"2023-12-15T22:58:49.439377+0000","last_clean":"2023-12-15T22:58:49.439377+0000","last_became_active":"2023-12-15T22:56:33.597320+0000","last_became_peered":"2023-12-15T22:56:33.597320+0000","last_unstale":"2023-12-15T22:58:49.439377+0000","last_undegraded":"2023-12-15T22:58:49.439377+0000","last_fullsized":"2023-12-15T22:58:49.439377+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"144'54","reported_seq":312,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-12-15T22:58:50.458407+0000","last_change":"2023-12-15T22:58:50.500003+0000","last_active":"2023-12-15T22:58:50.458407+0000","last_peered":"2023-12-15T22:58:50.458407+0000","last_clean":"2023-12-15T22:58:50.458407+0000","last_became_active":"2023-12-15T22:56:39.560980+0000","last_became_peered":"2023-12-15T22:56:39.560980+0000","last_unstale":"2023-12-15T22:58:50.458407+0000","last_undegraded":"2023-12-15T22:58:50.458407+0000","last_fullsized":"2023-12-15T22:58:50.458407+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5932904,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":29099,"num_write":30,"num_write_kb":9010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":217,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-12-15T22:58:50.466275+0000","last_change":"2023-12-15T22:56:33.648541+0000","last_active":"2023-12-15T22:58:50.466275+0000","last_peered":"2023-12-15T22:58:50.466275+0000","last_clean":"2023-12-15T22:58:50.466275+0000","last_became_active":"2023-12-15T22:56:33.643681+0000","last_became_peered":"2023-12-15T22:56:33.643681+0000","last_unstale":"2023-12-15T22:58:50.466275+0000","last_undegraded":"2023-12-15T22:58:50.466275+0000","last_fullsized":"2023-12-15T22:58:50.466275+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"143'103","reported_seq":350,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-15T22:58:49.451612+0000","last_change":"2023-12-15T22:58:49.537597+0000","last_active":"2023-12-15T22:58:49.451612+0000","last_peered":"2023-12-15T22:58:49.451612+0000","last_clean":"2023-12-15T22:58:49.451612+0000","last_became_active":"2023-12-15T22:56:39.506510+0000","last_became_peered":"2023-12-15T22:56:39.506510+0000","last_unstale":"2023-12-15T22:58:49.451612+0000","last_undegraded":"2023-12-15T22:58:49.451612+0000","last_fullsized":"2023-12-15T22:58:49.451612+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8252303,"num_objects":14,"num_object_clones":9,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":79,"num_read_kb":55525,"num_write":62,"num_write_kb":18137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"142'50","reported_seq":291,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-15T22:58:49.435525+0000","last_change":"2023-12-15T22:58:49.443380+0000","last_active":"2023-12-15T22:58:49.435525+0000","last_peered":"2023-12-15T22:58:49.435525+0000","last_clean":"2023-12-15T22:58:49.435525+0000","last_became_active":"2023-12-15T22:56:39.511159+0000","last_became_peered":"2023-12-15T22:56:39.511159+0000","last_unstale":"2023-12-15T22:58:49.435525+0000","last_undegraded":"2023-12-15T22:58:49.435525+0000","last_fullsized":"2023-12-15T22:58:49.435525+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3932838,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":11,"num_read_kb":6,"num_write":29,"num_write_kb":8954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":205,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-15T22:58:49.454488+0000","last_change":"2023-12-15T22:56:33.633890+0000","last_active":"2023-12-15T22:58:49.454488+0000","last_peered":"2023-12-15T22:58:49.454488+0000","last_clean":"2023-12-15T22:58:49.454488+0000","last_became_active":"2023-12-15T22:56:33.631134+0000","last_became_peered":"2023-12-15T22:56:33.631134+0000","last_unstale":"2023-12-15T22:58:49.454488+0000","last_undegraded":"2023-12-15T22:58:49.454488+0000","last_fullsized":"2023-12-15T22:58:49.454488+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"143'59","reported_seq":304,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-15T22:58:49.456775+0000","last_change":"2023-12-15T22:58:49.529843+0000","last_active":"2023-12-15T22:58:49.456775+0000","last_peered":"2023-12-15T22:58:49.456775+0000","last_clean":"2023-12-15T22:58:49.456775+0000","last_became_active":"2023-12-15T22:56:39.532730+0000","last_became_peered":"2023-12-15T22:56:39.532730+0000","last_unstale":"2023-12-15T22:58:49.456775+0000","last_undegraded":"2023-12-15T22:58:49.456775+0000","last_fullsized":"2023-12-15T22:58:49.456775+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2117715,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":43,"num_read_kb":38479,"num_write":31,"num_write_kb":9619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":215,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-12-15T22:58:50.454397+0000","last_change":"2023-12-15T22:56:33.655736+0000","last_active":"2023-12-15T22:58:50.454397+0000","last_peered":"2023-12-15T22:58:50.454397+0000","last_clean":"2023-12-15T22:58:50.454397+0000","last_became_active":"2023-12-15T22:56:33.652917+0000","last_became_peered":"2023-12-15T22:56:33.652917+0000","last_unstale":"2023-12-15T22:58:50.454397+0000","last_undegraded":"2023-12-15T22:58:50.454397+0000","last_fullsized":"2023-12-15T22:58:50.454397+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"144'92","reported_seq":358,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-12-15T22:58:50.453663+0000","last_change":"2023-12-15T22:58:50.535497+0000","last_active":"2023-12-15T22:58:50.453663+0000","last_peered":"2023-12-15T22:58:50.453663+0000","last_clean":"2023-12-15T22:58:50.453663+0000","last_became_active":"2023-12-15T22:56:39.557252+0000","last_became_peered":"2023-12-15T22:56:39.557252+0000","last_unstale":"2023-12-15T22:58:50.453663+0000","last_undegraded":"2023-12-15T22:58:50.453663+0000","last_fullsized":"2023-12-15T22:58:50.453663+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5764834,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":71,"num_read_kb":54271,"num_write":60,"num_write_kb":17442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":205,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-15T22:58:49.445359+0000","last_change":"2023-12-15T22:56:33.632348+0000","last_active":"2023-12-15T22:58:49.445359+0000","last_peered":"2023-12-15T22:58:49.445359+0000","last_clean":"2023-12-15T22:58:49.445359+0000","last_became_active":"2023-12-15T22:56:33.629022+0000","last_became_peered":"2023-12-15T22:56:33.629022+0000","last_unstale":"2023-12-15T22:58:49.445359+0000","last_undegraded":"2023-12-15T22:58:49.445359+0000","last_fullsized":"2023-12-15T22:58:49.445359+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"143'32","reported_seq":286,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-15T22:58:49.460840+0000","last_change":"2023-12-15T22:58:49.527148+0000","last_active":"2023-12-15T22:58:49.460840+0000","last_peered":"2023-12-15T22:58:49.460840+0000","last_clean":"2023-12-15T22:58:49.460840+0000","last_became_active":"2023-12-15T22:56:39.535145+0000","last_became_peered":"2023-12-15T22:56:39.535145+0000","last_unstale":"2023-12-15T22:58:49.460840+0000","last_undegraded":"2023-12-15T22:58:49.460840+0000","last_fullsized":"2023-12-15T22:58:49.460840+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2100908,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":29,"num_read_kb":29390,"num_write":17,"num_write_kb":5075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":211,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-15T22:58:49.434506+0000","last_change":"2023-12-15T22:56:33.627005+0000","last_active":"2023-12-15T22:58:49.434506+0000","last_peered":"2023-12-15T22:58:49.434506+0000","last_clean":"2023-12-15T22:58:49.434506+0000","last_became_active":"2023-12-15T22:56:33.622181+0000","last_became_peered":"2023-12-15T22:56:33.622181+0000","last_unstale":"2023-12-15T22:58:49.434506+0000","last_undegraded":"2023-12-15T22:58:49.434506+0000","last_fullsized":"2023-12-15T22:58:49.434506+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":108220537,"num_objects":182,"num_object_clones":120,"num_object_copies":546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":25,"num_read":1094,"num_read_kb":805036,"num_write":884,"num_write_kb":257264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1483,"ondisk_log_size":1483,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705804,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":127752,"kb_used_data":96768,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94244088,"statfs":{"total":96636764160,"available":96505946112,"internally_reserved":0,"allocated":99090432,"data_stored":112345834,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705803,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":103620,"kb_used_data":72636,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94268220,"statfs":{"total":96636764160,"available":96530657280,"internally_reserved":0,"allocated":74379264,"data_stored":85458944,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705803,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":128016,"kb_used_data":97032,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94243824,"statfs":{"total":96636764160,"available":96505675776,"internally_reserved":0,"allocated":99360768,"data_stored":114259070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705804,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":129920,"kb_used_data":98936,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94241920,"statfs":{"total":96636764160,"available":96503726080,"internally_reserved":0,"allocated":101310464,"data_stored":114630052,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-15T22:58:51.557 INFO:tasks.rados.rados.0.smithi071.stdout:535: finishing write tid 3 to smithi07183189-6 2023-12-15T22:58:51.557 INFO:tasks.rados.rados.0.smithi071.stdout:535: finishing write tid 4 to smithi07183189-6 2023-12-15T22:58:51.557 INFO:tasks.rados.rados.0.smithi071.stdout:535: oid 6 updating version 0 to 63 2023-12-15T22:58:51.557 INFO:tasks.rados.rados.0.smithi071.stdout:535: oid 6 updating version 63 to 64 2023-12-15T22:58:51.557 INFO:tasks.rados.rados.0.smithi071.stdout:535: oid 6 updating version 64 to 65 2023-12-15T22:58:51.557 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 6 v 65 (ObjNum 260 snap 70 seq_num 260) dirty exists 2023-12-15T22:58:51.557 INFO:tasks.rados.rados.0.smithi071.stdout:535: left oid 6 (ObjNum 260 snap 70 seq_num 260) 2023-12-15T22:58:51.557 INFO:tasks.rados.rados.0.smithi071.stdout:538: finishing write tid 1 to smithi07183189-24 2023-12-15T22:58:51.558 INFO:tasks.rados.rados.0.smithi071.stdout:538: finishing write tid 2 to smithi07183189-24 2023-12-15T22:58:51.558 INFO:tasks.rados.rados.0.smithi071.stdout:538: finishing write tid 3 to smithi07183189-24 2023-12-15T22:58:51.558 INFO:tasks.rados.rados.0.smithi071.stdout:538: finishing write tid 4 to smithi07183189-24 2023-12-15T22:58:51.558 INFO:tasks.rados.rados.0.smithi071.stdout:538: oid 24 updating version 0 to 52 2023-12-15T22:58:51.558 INFO:tasks.rados.rados.0.smithi071.stdout:538: oid 24 updating version 52 to 54 2023-12-15T22:58:51.558 INFO:tasks.rados.rados.0.smithi071.stdout:538: oid 24 version 54 is already newer than 53 2023-12-15T22:58:51.558 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 54 (ObjNum 262 snap 70 seq_num 262) dirty exists 2023-12-15T22:58:51.558 INFO:tasks.rados.rados.0.smithi071.stdout:538: left oid 24 (ObjNum 262 snap 70 seq_num 262) 2023-12-15T22:58:51.558 INFO:tasks.rados.rados.0.smithi071.stdout:535: done (3 left) 2023-12-15T22:58:51.558 INFO:tasks.rados.rados.0.smithi071.stdout:537: done (2 left) 2023-12-15T22:58:51.558 INFO:tasks.rados.rados.0.smithi071.stdout:538: done (1 left) 2023-12-15T22:58:51.558 INFO:tasks.rados.rados.0.smithi071.stdout:540: done (0 left) 2023-12-15T22:58:51.558 INFO:tasks.rados.rados.0.smithi071.stdout:541: write (excl) oid 44 current snap is 70 2023-12-15T22:58:51.558 INFO:tasks.rados.rados.0.smithi071.stdout:541: seq_num 264 ranges {437048=621738,1832305=404425} 2023-12-15T22:58:51.562 INFO:tasks.rados.rados.0.smithi071.stdout:541: writing smithi07183189-44 from 437048 to 1058786 tid 1 2023-12-15T22:58:51.565 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:58:51.569 INFO:tasks.rados.rados.0.smithi071.stdout:541: writing smithi07183189-44 from 1832305 to 2236730 tid 2 2023-12-15T22:58:51.569 INFO:tasks.rados.rados.0.smithi071.stdout:542: write oid 31 current snap is 70 2023-12-15T22:58:51.569 INFO:tasks.rados.rados.0.smithi071.stdout:542: seq_num 265 ranges {453855=696987,1890787=548083} 2023-12-15T22:58:51.576 INFO:tasks.rados.rados.0.smithi071.stdout:542: writing smithi07183189-31 from 453855 to 1150842 tid 1 2023-12-15T22:58:51.583 INFO:tasks.rados.rados.0.smithi071.stdout:542: writing smithi07183189-31 from 1890787 to 2438870 tid 2 2023-12-15T22:58:51.583 INFO:tasks.rados.rados.0.smithi071.stdout:543: write oid 21 current snap is 70 2023-12-15T22:58:51.584 INFO:tasks.rados.rados.0.smithi071.stdout:543: seq_num 266 ranges {470662=772236,1832916=408094} 2023-12-15T22:58:51.590 INFO:tasks.rados.rados.0.smithi071.stdout:543: writing smithi07183189-21 from 470662 to 1242898 tid 1 2023-12-15T22:58:51.596 INFO:tasks.rados.rados.0.smithi071.stdout:543: writing smithi07183189-21 from 1832916 to 2241010 tid 2 2023-12-15T22:58:51.596 INFO:tasks.rados.rados.0.smithi071.stdout:544: delete oid 45 current snap is 70 2023-12-15T22:58:51.596 INFO:tasks.rados.rados.0.smithi071.stdout:541: finishing write tid 1 to smithi07183189-44 2023-12-15T22:58:51.599 INFO:tasks.rados.rados.0.smithi071.stdout:544: done (3 left) 2023-12-15T22:58:51.599 INFO:tasks.rados.rados.0.smithi071.stdout:545: write oid 7 current snap is 70 2023-12-15T22:58:51.599 INFO:tasks.rados.rados.0.smithi071.stdout:545: seq_num 268 ranges {504276=639087,1666233=695410} 2023-12-15T22:58:51.605 INFO:tasks.rados.rados.0.smithi071.stdout:545: writing smithi07183189-7 from 504276 to 1143363 tid 1 2023-12-15T22:58:51.613 INFO:tasks.rados.rados.0.smithi071.stdout:545: writing smithi07183189-7 from 1666233 to 2361643 tid 2 2023-12-15T22:58:51.613 INFO:tasks.rados.rados.0.smithi071.stdout:546: read oid 47 snap 63 2023-12-15T22:58:51.613 INFO:tasks.rados.rados.0.smithi071.stdout:546: expect (ObjNum 193 snap 38 seq_num 193) 2023-12-15T22:58:51.613 INFO:tasks.rados.rados.0.smithi071.stdout:547: write oid 40 current snap is 70 2023-12-15T22:58:51.613 INFO:tasks.rados.rados.0.smithi071.stdout:547: seq_num 269 ranges {521083=714336,1724715=555421} 2023-12-15T22:58:51.620 INFO:tasks.rados.rados.0.smithi071.stdout:547: writing smithi07183189-40 from 521083 to 1235419 tid 1 2023-12-15T22:58:51.627 INFO:tasks.rados.rados.0.smithi071.stdout:547: writing smithi07183189-40 from 1724715 to 2280136 tid 2 2023-12-15T22:58:51.627 INFO:tasks.rados.rados.0.smithi071.stdout:548: snap_remove snap 66 2023-12-15T22:58:51.678 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:51.676+0000 7f430706d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:51.681 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:51.681+0000 7f430706d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:52.256 INFO:tasks.rados.rados.0.smithi071.stdout:548: done (6 left) 2023-12-15T22:58:52.256 INFO:tasks.rados.rados.0.smithi071.stdout:549: write oid 11 current snap is 70 2023-12-15T22:58:52.256 INFO:tasks.rados.rados.0.smithi071.stdout:541: finishing write tid 2 to smithi07183189-44 2023-12-15T22:58:52.256 INFO:tasks.rados.rados.0.smithi071.stdout:549: seq_num 270 ranges {537890=789585,1783197=699079} 2023-12-15T22:58:52.263 INFO:tasks.rados.rados.0.smithi071.stdout:549: writing smithi07183189-11 from 537890 to 1327475 tid 1 2023-12-15T22:58:52.270 INFO:tasks.rados.rados.0.smithi071.stdout:549: writing smithi07183189-11 from 1783197 to 2482276 tid 2 2023-12-15T22:58:52.270 INFO:tasks.rados.rados.0.smithi071.stdout:550: write (excl) oid 9 current snap is 70 2023-12-15T22:58:52.270 INFO:tasks.rados.rados.0.smithi071.stdout:550: seq_num 271 ranges {554697=464834,1725326=559090} 2023-12-15T22:58:52.276 INFO:tasks.rados.rados.0.smithi071.stdout:550: writing smithi07183189-9 from 554697 to 1019531 tid 1 2023-12-15T22:58:52.284 INFO:tasks.rados.rados.0.smithi071.stdout:550: writing smithi07183189-9 from 1725326 to 2284416 tid 2 2023-12-15T22:58:52.284 INFO:tasks.rados.rados.0.smithi071.stdout:551: snap_remove snap 46 2023-12-15T22:58:53.262 INFO:tasks.rados.rados.0.smithi071.stdout:551: done (8 left) 2023-12-15T22:58:53.262 INFO:tasks.rados.rados.0.smithi071.stdout:552: write (excl) oid 35 current snap is 70 2023-12-15T22:58:53.262 INFO:tasks.rados.rados.0.smithi071.stdout:552: seq_num 272 ranges {571504=540083,1783808=702748} 2023-12-15T22:58:53.268 INFO:tasks.rados.rados.0.smithi071.stdout:552: writing smithi07183189-35 from 571504 to 1111587 tid 1 2023-12-15T22:58:53.277 INFO:tasks.rados.rados.0.smithi071.stdout:552: writing smithi07183189-35 from 1783808 to 2486556 tid 2 2023-12-15T22:58:53.277 INFO:tasks.rados.rados.0.smithi071.stdout:553: read oid 46 snap 58 2023-12-15T22:58:53.277 INFO:tasks.rados.rados.0.smithi071.stdout:553: expect deleted 2023-12-15T22:58:53.277 INFO:tasks.rados.rados.0.smithi071.stdout:554: read oid 45 snap -1 2023-12-15T22:58:53.277 INFO:tasks.rados.rados.0.smithi071.stdout:554: expect deleted 2023-12-15T22:58:53.277 INFO:tasks.rados.rados.0.smithi071.stdout:555: delete oid 39 current snap is 70 2023-12-15T22:58:53.277 INFO:tasks.rados.rados.0.smithi071.stdout:541: finishing write tid 3 to smithi07183189-44 2023-12-15T22:58:53.277 INFO:tasks.rados.rados.0.smithi071.stdout:541: finishing write tid 4 to smithi07183189-44 2023-12-15T22:58:53.277 INFO:tasks.rados.rados.0.smithi071.stdout:541: oid 44 updating version 0 to 63 2023-12-15T22:58:53.278 INFO:tasks.rados.rados.0.smithi071.stdout:541: oid 44 updating version 63 to 64 2023-12-15T22:58:53.278 INFO:tasks.rados.rados.0.smithi071.stdout:541: oid 44 updating version 64 to 65 2023-12-15T22:58:53.278 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 65 (ObjNum 264 snap 70 seq_num 264) dirty exists 2023-12-15T22:58:53.278 INFO:tasks.rados.rados.0.smithi071.stdout:541: left oid 44 (ObjNum 264 snap 70 seq_num 264) 2023-12-15T22:58:53.278 INFO:tasks.rados.rados.0.smithi071.stdout:546: expect (ObjNum 193 snap 38 seq_num 193) 2023-12-15T22:58:53.476 INFO:tasks.rados.rados.0.smithi071.stdout:547: finishing write tid 1 to smithi07183189-40 2023-12-15T22:58:53.476 INFO:tasks.rados.rados.0.smithi071.stdout:543: finishing write tid 1 to smithi07183189-21 2023-12-15T22:58:53.476 INFO:tasks.rados.rados.0.smithi071.stdout:547: finishing write tid 2 to smithi07183189-40 2023-12-15T22:58:53.476 INFO:tasks.rados.rados.0.smithi071.stdout:545: finishing write tid 1 to smithi07183189-7 2023-12-15T22:58:53.477 INFO:tasks.rados.rados.0.smithi071.stdout:541: done (11 left) 2023-12-15T22:58:53.477 INFO:tasks.rados.rados.0.smithi071.stdout:546: done (10 left) 2023-12-15T22:58:53.477 INFO:tasks.rados.rados.0.smithi071.stdout:555: done (9 left) 2023-12-15T22:58:53.477 INFO:tasks.rados.rados.0.smithi071.stdout:556: snap_remove snap 57 2023-12-15T22:58:53.855 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:58:53.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:53.957+0000 7f731939e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:53.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:53.962+0000 7f731939e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:54.431 INFO:tasks.rados.rados.0.smithi071.stdout:556: done (9 left) 2023-12-15T22:58:54.431 INFO:tasks.rados.rados.0.smithi071.stdout:557: read oid 10 snap -1 2023-12-15T22:58:54.431 INFO:tasks.rados.rados.0.smithi071.stdout:557: expect (ObjNum 147 snap 20 seq_num 147) 2023-12-15T22:58:54.432 INFO:tasks.rados.rados.0.smithi071.stdout:558: snap_remove snap 68 2023-12-15T22:58:55.437 INFO:tasks.rados.rados.0.smithi071.stdout:558: done (10 left) 2023-12-15T22:58:55.438 INFO:tasks.rados.rados.0.smithi071.stdout:559: read oid 39 snap -1 2023-12-15T22:58:55.438 INFO:tasks.rados.rados.0.smithi071.stdout:559: expect deleted 2023-12-15T22:58:55.438 INFO:tasks.rados.rados.0.smithi071.stdout:560: write oid 25 current snap is 70 2023-12-15T22:58:55.438 INFO:tasks.rados.rados.0.smithi071.stdout:542: finishing write tid 1 to smithi07183189-31 2023-12-15T22:58:55.438 INFO:tasks.rados.rados.0.smithi071.stdout:547: finishing write tid 3 to smithi07183189-40 2023-12-15T22:58:55.438 INFO:tasks.rados.rados.0.smithi071.stdout:547: finishing write tid 4 to smithi07183189-40 2023-12-15T22:58:55.438 INFO:tasks.rados.rados.0.smithi071.stdout:547: oid 40 updating version 0 to 313 2023-12-15T22:58:55.438 INFO:tasks.rados.rados.0.smithi071.stdout:547: oid 40 updating version 313 to 314 2023-12-15T22:58:55.438 INFO:tasks.rados.rados.0.smithi071.stdout:547: oid 40 updating version 314 to 315 2023-12-15T22:58:55.438 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 40 v 315 (ObjNum 269 snap 70 seq_num 269) dirty exists 2023-12-15T22:58:55.438 INFO:tasks.rados.rados.0.smithi071.stdout:547: left oid 40 (ObjNum 269 snap 70 seq_num 269) 2023-12-15T22:58:55.438 INFO:tasks.rados.rados.0.smithi071.stdout:543: finishing write tid 2 to smithi07183189-21 2023-12-15T22:58:55.438 INFO:tasks.rados.rados.0.smithi071.stdout:545: finishing write tid 2 to smithi07183189-7 2023-12-15T22:58:55.438 INFO:tasks.rados.rados.0.smithi071.stdout:545: finishing write tid 3 to smithi07183189-7 2023-12-15T22:58:55.439 INFO:tasks.rados.rados.0.smithi071.stdout:545: finishing write tid 4 to smithi07183189-7 2023-12-15T22:58:55.439 INFO:tasks.rados.rados.0.smithi071.stdout:545: oid 7 updating version 0 to 184 2023-12-15T22:58:55.439 INFO:tasks.rados.rados.0.smithi071.stdout:545: oid 7 version 184 is already newer than 182 2023-12-15T22:58:55.439 INFO:tasks.rados.rados.0.smithi071.stdout:545: oid 7 version 184 is already newer than 184 2023-12-15T22:58:55.439 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 184 (ObjNum 268 snap 70 seq_num 268) dirty exists 2023-12-15T22:58:55.439 INFO:tasks.rados.rados.0.smithi071.stdout:545: left oid 7 (ObjNum 268 snap 70 seq_num 268) 2023-12-15T22:58:55.439 INFO:tasks.rados.rados.0.smithi071.stdout:542: finishing write tid 2 to smithi07183189-31 2023-12-15T22:58:55.439 INFO:tasks.rados.rados.0.smithi071.stdout:542: finishing write tid 3 to smithi07183189-31 2023-12-15T22:58:55.439 INFO:tasks.rados.rados.0.smithi071.stdout:542: finishing write tid 4 to smithi07183189-31 2023-12-15T22:58:55.439 INFO:tasks.rados.rados.0.smithi071.stdout:542: oid 31 updating version 0 to 100 2023-12-15T22:58:55.439 INFO:tasks.rados.rados.0.smithi071.stdout:542: oid 31 updating version 100 to 102 2023-12-15T22:58:55.439 INFO:tasks.rados.rados.0.smithi071.stdout:542: oid 31 version 102 is already newer than 102 2023-12-15T22:58:55.439 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 102 (ObjNum 265 snap 70 seq_num 265) dirty exists 2023-12-15T22:58:55.439 INFO:tasks.rados.rados.0.smithi071.stdout:542: left oid 31 (ObjNum 265 snap 70 seq_num 265) 2023-12-15T22:58:55.439 INFO:tasks.rados.rados.0.smithi071.stdout:543: finishing write tid 3 to smithi07183189-21 2023-12-15T22:58:55.439 INFO:tasks.rados.rados.0.smithi071.stdout:543: finishing write tid 4 to smithi07183189-21 2023-12-15T22:58:55.440 INFO:tasks.rados.rados.0.smithi071.stdout:543: oid 21 updating version 0 to 158 2023-12-15T22:58:55.440 INFO:tasks.rados.rados.0.smithi071.stdout:543: oid 21 updating version 158 to 159 2023-12-15T22:58:55.440 INFO:tasks.rados.rados.0.smithi071.stdout:543: oid 21 version 159 is already newer than 157 2023-12-15T22:58:55.440 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 159 (ObjNum 266 snap 70 seq_num 266) dirty exists 2023-12-15T22:58:55.440 INFO:tasks.rados.rados.0.smithi071.stdout:543: left oid 21 (ObjNum 266 snap 70 seq_num 266) 2023-12-15T22:58:55.440 INFO:tasks.rados.rados.0.smithi071.stdout:549: finishing write tid 1 to smithi07183189-11 2023-12-15T22:58:55.440 INFO:tasks.rados.rados.0.smithi071.stdout:550: finishing write tid 1 to smithi07183189-9 2023-12-15T22:58:55.440 INFO:tasks.rados.rados.0.smithi071.stdout:549: finishing write tid 2 to smithi07183189-11 2023-12-15T22:58:55.440 INFO:tasks.rados.rados.0.smithi071.stdout:549: finishing write tid 3 to smithi07183189-11 2023-12-15T22:58:55.440 INFO:tasks.rados.rados.0.smithi071.stdout:549: finishing write tid 4 to smithi07183189-11 2023-12-15T22:58:55.440 INFO:tasks.rados.rados.0.smithi071.stdout:549: oid 11 updating version 0 to 106 2023-12-15T22:58:55.440 INFO:tasks.rados.rados.0.smithi071.stdout:549: oid 11 updating version 106 to 108 2023-12-15T22:58:55.440 INFO:tasks.rados.rados.0.smithi071.stdout:549: oid 11 version 108 is already newer than 108 2023-12-15T22:58:55.441 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 11 v 108 (ObjNum 270 snap 70 seq_num 270) dirty exists 2023-12-15T22:58:55.441 INFO:tasks.rados.rados.0.smithi071.stdout:549: left oid 11 (ObjNum 270 snap 70 seq_num 270) 2023-12-15T22:58:55.441 INFO:tasks.rados.rados.0.smithi071.stdout:550: finishing write tid 2 to smithi07183189-9 2023-12-15T22:58:55.441 INFO:tasks.rados.rados.0.smithi071.stdout:550: finishing write tid 3 to smithi07183189-9 2023-12-15T22:58:55.441 INFO:tasks.rados.rados.0.smithi071.stdout:550: finishing write tid 4 to smithi07183189-9 2023-12-15T22:58:55.441 INFO:tasks.rados.rados.0.smithi071.stdout:550: oid 9 updating version 0 to 131 2023-12-15T22:58:55.441 INFO:tasks.rados.rados.0.smithi071.stdout:550: oid 9 version 131 is already newer than 128 2023-12-15T22:58:55.441 INFO:tasks.rados.rados.0.smithi071.stdout:550: oid 9 version 131 is already newer than 127 2023-12-15T22:58:55.441 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 131 (ObjNum 271 snap 70 seq_num 271) dirty exists 2023-12-15T22:58:55.441 INFO:tasks.rados.rados.0.smithi071.stdout:550: left oid 9 (ObjNum 271 snap 70 seq_num 271) 2023-12-15T22:58:55.441 INFO:tasks.rados.rados.0.smithi071.stdout:552: finishing write tid 1 to smithi07183189-35 2023-12-15T22:58:55.441 INFO:tasks.rados.rados.0.smithi071.stdout:552: finishing write tid 2 to smithi07183189-35 2023-12-15T22:58:55.441 INFO:tasks.rados.rados.0.smithi071.stdout:560: seq_num 274 ranges {605118=406934,1617125=706417} 2023-12-15T22:58:55.442 INFO:tasks.rados.rados.0.smithi071.stdout:560: writing smithi07183189-25 from 605118 to 1012052 tid 1 2023-12-15T22:58:55.451 INFO:tasks.rados.rados.0.smithi071.stdout:560: writing smithi07183189-25 from 1617125 to 2323542 tid 2 2023-12-15T22:58:55.451 INFO:tasks.rados.rados.0.smithi071.stdout:542: done (11 left) 2023-12-15T22:58:55.451 INFO:tasks.rados.rados.0.smithi071.stdout:543: done (10 left) 2023-12-15T22:58:55.451 INFO:tasks.rados.rados.0.smithi071.stdout:545: done (9 left) 2023-12-15T22:58:55.451 INFO:tasks.rados.rados.0.smithi071.stdout:547: done (8 left) 2023-12-15T22:58:55.451 INFO:tasks.rados.rados.0.smithi071.stdout:549: done (7 left) 2023-12-15T22:58:55.451 INFO:tasks.rados.rados.0.smithi071.stdout:550: done (6 left) 2023-12-15T22:58:55.451 INFO:tasks.rados.rados.0.smithi071.stdout:553: done (5 left) 2023-12-15T22:58:55.451 INFO:tasks.rados.rados.0.smithi071.stdout:554: done (4 left) 2023-12-15T22:58:55.451 INFO:tasks.rados.rados.0.smithi071.stdout:561: snap_remove snap 53 2023-12-15T22:58:56.029 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:58:56.135 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:56.133+0000 7fcc9f311640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:56.138 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:56.138+0000 7fcc9f311640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:56.438 INFO:tasks.rados.rados.0.smithi071.stdout:561: done (4 left) 2023-12-15T22:58:56.439 INFO:tasks.rados.rados.0.smithi071.stdout:562: snap_create 2023-12-15T22:58:56.439 INFO:tasks.rados.rados.0.smithi071.stdout:552: finishing write tid 3 to smithi07183189-35 2023-12-15T22:58:56.439 INFO:tasks.rados.rados.0.smithi071.stdout:552: finishing write tid 4 to smithi07183189-35 2023-12-15T22:58:56.439 INFO:tasks.rados.rados.0.smithi071.stdout:552: oid 35 updating version 0 to 135 2023-12-15T22:58:56.439 INFO:tasks.rados.rados.0.smithi071.stdout:552: oid 35 updating version 135 to 136 2023-12-15T22:58:56.439 INFO:tasks.rados.rados.0.smithi071.stdout:552: oid 35 updating version 136 to 137 2023-12-15T22:58:56.439 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 137 (ObjNum 272 snap 70 seq_num 272) dirty exists 2023-12-15T22:58:56.439 INFO:tasks.rados.rados.0.smithi071.stdout:552: left oid 35 (ObjNum 272 snap 70 seq_num 272) 2023-12-15T22:58:56.439 INFO:tasks.rados.rados.0.smithi071.stdout:557: expect (ObjNum 147 snap 20 seq_num 147) 2023-12-15T22:58:56.528 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T22:58:56.528 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T22:58:56.528 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-12-15T22:58:56.528 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-12-15T22:58:56.543 INFO:tasks.rados.rados.0.smithi071.stdout:560: finishing write tid 1 to smithi07183189-25 2023-12-15T22:58:56.543 INFO:tasks.rados.rados.0.smithi071.stdout:560: finishing write tid 2 to smithi07183189-25 2023-12-15T22:58:56.543 INFO:tasks.rados.rados.0.smithi071.stdout:560: finishing write tid 3 to smithi07183189-25 2023-12-15T22:58:56.543 INFO:tasks.rados.rados.0.smithi071.stdout:560: finishing write tid 4 to smithi07183189-25 2023-12-15T22:58:56.543 INFO:tasks.rados.rados.0.smithi071.stdout:560: oid 25 updating version 0 to 195 2023-12-15T22:58:56.543 INFO:tasks.rados.rados.0.smithi071.stdout:560: oid 25 updating version 195 to 196 2023-12-15T22:58:56.543 INFO:tasks.rados.rados.0.smithi071.stdout:560: oid 25 version 196 is already newer than 192 2023-12-15T22:58:56.543 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 25 v 196 (ObjNum 274 snap 70 seq_num 274) dirty exists 2023-12-15T22:58:56.543 INFO:tasks.rados.rados.0.smithi071.stdout:560: left oid 25 (ObjNum 274 snap 70 seq_num 274) 2023-12-15T22:58:56.641 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:56.636+0000 7fea0779e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:56.645 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:56.645+0000 7fea0779e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:56.682 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:58:56.682 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:58:56.682 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:58:56.682 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:58:56.682 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:58:56.682 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:58:56.682 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:58:56.682 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:58:56.682 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:58:56.682 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:58:56.682 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:58:56.683 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:58:56.685 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:58:56.685 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:58:56.685 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-12-15T22:58:56.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:56.827+0000 7fa49655e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:56.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:56.834+0000 7fa49655e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:56.846 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:58:56.846 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:58:56.847 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:58:56.847 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:58:56.847 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:58:56.847 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:58:56.847 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:58:56.847 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:58:56.847 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:58:56.847 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:58:56.847 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:58:56.847 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:58:56.849 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:58:56.850 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:58:56.850 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-12-15T22:58:56.956 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:56.951+0000 7f31b715e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:56.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:56.959+0000 7f31b715e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:56.971 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:58:56.971 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:58:56.971 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:58:56.971 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:58:56.971 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:58:56.971 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:58:56.971 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:58:56.971 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:58:56.971 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:58:56.971 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:58:56.972 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:58:56.972 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:58:56.973 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:58:56.974 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:58:56.974 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-12-15T22:58:57.081 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:57.077+0000 7ff395e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:57.084 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:57.084+0000 7ff395e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:57.096 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:58:57.096 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:58:57.096 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:58:57.096 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:58:57.097 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:58:57.097 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:58:57.097 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:58:57.097 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:58:57.097 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:58:57.097 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:58:57.097 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:58:57.097 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:58:57.099 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:58:57.099 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:58:57.448 INFO:tasks.rados.rados.0.smithi071.stdout:552: done (4 left) 2023-12-15T22:58:57.448 INFO:tasks.rados.rados.0.smithi071.stdout:557: done (3 left) 2023-12-15T22:58:57.448 INFO:tasks.rados.rados.0.smithi071.stdout:559: done (2 left) 2023-12-15T22:58:57.448 INFO:tasks.rados.rados.0.smithi071.stdout:560: done (1 left) 2023-12-15T22:58:57.448 INFO:tasks.rados.rados.0.smithi071.stdout:562: done (0 left) 2023-12-15T22:58:57.448 INFO:tasks.rados.rados.0.smithi071.stdout:563: write oid 6 current snap is 71 2023-12-15T22:58:57.448 INFO:tasks.rados.rados.0.smithi071.stdout:563: seq_num 275 ranges {621925=482183,1559254=566428} 2023-12-15T22:58:57.455 INFO:tasks.rados.rados.0.smithi071.stdout:563: writing smithi07183189-6 from 621925 to 1104108 tid 1 2023-12-15T22:58:57.463 INFO:tasks.rados.rados.0.smithi071.stdout:563: writing smithi07183189-6 from 1559254 to 2125682 tid 2 2023-12-15T22:58:57.463 INFO:tasks.rados.rados.0.smithi071.stdout:564: snap_create 2023-12-15T22:58:57.541 INFO:tasks.rados.rados.0.smithi071.stdout:563: finishing write tid 1 to smithi07183189-6 2023-12-15T22:58:57.567 INFO:tasks.rados.rados.0.smithi071.stdout:563: finishing write tid 2 to smithi07183189-6 2023-12-15T22:58:57.574 INFO:tasks.rados.rados.0.smithi071.stdout:563: finishing write tid 3 to smithi07183189-6 2023-12-15T22:58:57.576 INFO:tasks.rados.rados.0.smithi071.stdout:563: finishing write tid 4 to smithi07183189-6 2023-12-15T22:58:57.576 INFO:tasks.rados.rados.0.smithi071.stdout:563: oid 6 updating version 0 to 77 2023-12-15T22:58:57.576 INFO:tasks.rados.rados.0.smithi071.stdout:563: oid 6 updating version 77 to 79 2023-12-15T22:58:57.576 INFO:tasks.rados.rados.0.smithi071.stdout:563: oid 6 version 79 is already newer than 78 2023-12-15T22:58:57.576 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 6 v 79 (ObjNum 275 snap 71 seq_num 275) dirty exists 2023-12-15T22:58:57.576 INFO:tasks.rados.rados.0.smithi071.stdout:563: left oid 6 (ObjNum 275 snap 71 seq_num 275) 2023-12-15T22:58:58.205 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:58:58.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:58.308+0000 7f646255e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:58.312 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:58:58.312+0000 7f646255e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:58:58.450 INFO:tasks.rados.rados.0.smithi071.stdout:563: done (1 left) 2023-12-15T22:58:58.450 INFO:tasks.rados.rados.0.smithi071.stdout:564: done (0 left) 2023-12-15T22:58:58.451 INFO:tasks.rados.rados.0.smithi071.stdout:565: write oid 22 current snap is 72 2023-12-15T22:58:58.451 INFO:tasks.rados.rados.0.smithi071.stdout:565: seq_num 276 ranges {638732=557432,1617736=710086} 2023-12-15T22:58:58.458 INFO:tasks.rados.rados.0.smithi071.stdout:565: writing smithi07183189-22 from 638732 to 1196164 tid 1 2023-12-15T22:58:58.467 INFO:tasks.rados.rados.0.smithi071.stdout:565: writing smithi07183189-22 from 1617736 to 2327822 tid 2 2023-12-15T22:58:58.589 INFO:tasks.rados.rados.0.smithi071.stdout:566: snap_remove snap 42 2023-12-15T22:58:59.454 INFO:tasks.rados.rados.0.smithi071.stdout:566: done (1 left) 2023-12-15T22:58:59.454 INFO:tasks.rados.rados.0.smithi071.stdout:567: snap_create 2023-12-15T22:58:59.454 INFO:tasks.rados.rados.0.smithi071.stdout:565: finishing write tid 1 to smithi07183189-22 2023-12-15T22:58:59.454 INFO:tasks.rados.rados.0.smithi071.stdout:565: finishing write tid 2 to smithi07183189-22 2023-12-15T22:58:59.454 INFO:tasks.rados.rados.0.smithi071.stdout:565: finishing write tid 3 to smithi07183189-22 2023-12-15T22:58:59.454 INFO:tasks.rados.rados.0.smithi071.stdout:565: finishing write tid 4 to smithi07183189-22 2023-12-15T22:58:59.454 INFO:tasks.rados.rados.0.smithi071.stdout:565: oid 22 updating version 0 to 128 2023-12-15T22:58:59.454 INFO:tasks.rados.rados.0.smithi071.stdout:565: oid 22 version 128 is already newer than 127 2023-12-15T22:58:59.454 INFO:tasks.rados.rados.0.smithi071.stdout:565: oid 22 version 128 is already newer than 126 2023-12-15T22:58:59.454 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 128 (ObjNum 276 snap 72 seq_num 276) dirty exists 2023-12-15T22:58:59.454 INFO:tasks.rados.rados.0.smithi071.stdout:565: left oid 22 (ObjNum 276 snap 72 seq_num 276) 2023-12-15T22:59:00.381 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:59:00.454 INFO:tasks.rados.rados.0.smithi071.stdout:565: done (1 left) 2023-12-15T22:59:00.454 INFO:tasks.rados.rados.0.smithi071.stdout:567: done (0 left) 2023-12-15T22:59:00.454 INFO:tasks.rados.rados.0.smithi071.stdout:568: snap_create 2023-12-15T22:59:00.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:00.496+0000 7fb4ba71e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:00.500 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:00.500+0000 7fb4ba71e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:01.456 INFO:tasks.rados.rados.0.smithi071.stdout:568: done (0 left) 2023-12-15T22:59:01.456 INFO:tasks.rados.rados.0.smithi071.stdout:569: snap_remove snap 69 2023-12-15T22:59:02.100 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T22:59:02.100 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T22:59:02.101 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T22:59:02.218 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:02.215+0000 7f3fc779e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:02.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:02.222+0000 7f3fc779e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:02.255 INFO:tasks.rados.rados.0.smithi071.stdout:569: done (0 left) 2023-12-15T22:59:02.255 INFO:tasks.rados.rados.0.smithi071.stdout:570: write oid 46 current snap is 74 2023-12-15T22:59:02.255 INFO:tasks.rados.rados.0.smithi071.stdout:570: seq_num 277 ranges {655539=632681,2076218=570097} 2023-12-15T22:59:02.263 INFO:tasks.rados.rados.0.smithi071.stdout:570: writing smithi07183189-46 from 655539 to 1288220 tid 1 2023-12-15T22:59:02.272 INFO:tasks.rados.rados.0.smithi071.stdout:570: writing smithi07183189-46 from 2076218 to 2646315 tid 2 2023-12-15T22:59:02.272 INFO:tasks.rados.rados.0.smithi071.stdout:571: snap_remove snap 50 2023-12-15T22:59:02.438 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:59:02.438 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-12-15T22:59:02.449 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2023-12-15T22:59:02.253733+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":107634712,"num_objects":195,"num_object_clones":131,"num_object_copies":585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":24,"num_read":1242,"num_read_kb":831167,"num_write":1077,"num_write_kb":274828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1887,"ondisk_log_size":1887,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":506044,"kb_used_data":382108,"kb_used_omap":0,"kb_used_meta":123904,"kb_avail":376981316,"statfs":{"total":386547056640,"available":386028867584,"internally_reserved":0,"allocated":391278592,"data_stored":441199903,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":126877696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":941225,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":12,"num_write_kb":3352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.803743"},"pg_stats":[{"pgid":"3.f","version":"73'24","reported_seq":329,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-12-15T22:58:58.543290+0000","last_change":"2023-12-15T22:58:56.645767+0000","last_active":"2023-12-15T22:58:58.543290+0000","last_peered":"2023-12-15T22:58:58.543290+0000","last_clean":"2023-12-15T22:58:58.543290+0000","last_became_active":"2023-12-15T22:56:39.503797+0000","last_became_peered":"2023-12-15T22:56:39.503797+0000","last_unstale":"2023-12-15T22:58:58.543290+0000","last_undegraded":"2023-12-15T22:58:58.543290+0000","last_fullsized":"2023-12-15T22:58:58.543290+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4941225,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":50,"num_read_kb":39853,"num_write":17,"num_write_kb":4759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":278,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-12-15T22:58:58.475338+0000","last_change":"2023-12-15T22:58:56.654654+0000","last_active":"2023-12-15T22:58:58.475338+0000","last_peered":"2023-12-15T22:58:58.475338+0000","last_clean":"2023-12-15T22:58:58.475338+0000","last_became_active":"2023-12-15T22:56:39.552856+0000","last_became_peered":"2023-12-15T22:56:39.552856+0000","last_unstale":"2023-12-15T22:58:58.475338+0000","last_undegraded":"2023-12-15T22:58:58.475338+0000","last_fullsized":"2023-12-15T22:58:58.475338+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"151'202","reported_seq":476,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-12-15T22:58:58.457395+0000","last_change":"2023-12-15T22:58:57.549067+0000","last_active":"2023-12-15T22:58:58.457395+0000","last_peered":"2023-12-15T22:58:58.457395+0000","last_clean":"2023-12-15T22:58:58.457395+0000","last_became_active":"2023-12-15T22:56:39.495490+0000","last_became_peered":"2023-12-15T22:56:39.495490+0000","last_unstale":"2023-12-15T22:58:58.457395+0000","last_undegraded":"2023-12-15T22:58:58.457395+0000","last_fullsized":"2023-12-15T22:58:58.457395+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":202,"log_dups_size":0,"ondisk_log_size":202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15563348,"num_objects":24,"num_object_clones":15,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":172,"num_read_kb":141222,"num_write":127,"num_write_kb":36642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"150'149","reported_seq":410,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-12-15T22:58:58.541133+0000","last_change":"2023-12-15T22:58:56.702204+0000","last_active":"2023-12-15T22:58:58.541133+0000","last_peered":"2023-12-15T22:58:58.541133+0000","last_clean":"2023-12-15T22:58:58.541133+0000","last_became_active":"2023-12-15T22:56:39.579234+0000","last_became_peered":"2023-12-15T22:56:39.579234+0000","last_unstale":"2023-12-15T22:58:58.541133+0000","last_undegraded":"2023-12-15T22:58:58.541133+0000","last_fullsized":"2023-12-15T22:58:58.541133+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8470794,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":100,"num_read_kb":73572,"num_write":86,"num_write_kb":23729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"153'89","reported_seq":353,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-12-15T22:59:00.462092+0000","last_change":"2023-12-15T22:58:59.527284+0000","last_active":"2023-12-15T22:59:00.462092+0000","last_peered":"2023-12-15T22:59:00.462092+0000","last_clean":"2023-12-15T22:59:00.462092+0000","last_became_active":"2023-12-15T22:56:39.572176+0000","last_became_peered":"2023-12-15T22:56:39.572176+0000","last_unstale":"2023-12-15T22:59:00.462092+0000","last_undegraded":"2023-12-15T22:59:00.462092+0000","last_fullsized":"2023-12-15T22:59:00.462092+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3075681,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":45,"num_read_kb":23791,"num_write":41,"num_write_kb":12228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"150'90","reported_seq":347,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-12-15T22:58:58.541086+0000","last_change":"2023-12-15T22:58:56.695728+0000","last_active":"2023-12-15T22:58:58.541086+0000","last_peered":"2023-12-15T22:58:58.541086+0000","last_clean":"2023-12-15T22:58:58.541086+0000","last_became_active":"2023-12-15T22:56:39.516361+0000","last_became_peered":"2023-12-15T22:56:39.516361+0000","last_unstale":"2023-12-15T22:58:58.541086+0000","last_undegraded":"2023-12-15T22:58:58.541086+0000","last_fullsized":"2023-12-15T22:58:58.541086+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4453921,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":36,"num_read_kb":25582,"num_write":47,"num_write_kb":13134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"150'345","reported_seq":515,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-12-15T22:58:58.548565+0000","last_change":"2023-12-15T22:58:56.721989+0000","last_active":"2023-12-15T22:58:58.548565+0000","last_peered":"2023-12-15T22:58:58.548565+0000","last_clean":"2023-12-15T22:58:58.548565+0000","last_became_active":"2023-12-15T22:56:39.519748+0000","last_became_peered":"2023-12-15T22:56:39.519748+0000","last_unstale":"2023-12-15T22:58:58.548565+0000","last_undegraded":"2023-12-15T22:58:58.548565+0000","last_fullsized":"2023-12-15T22:58:58.548565+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15647350,"num_objects":38,"num_object_clones":28,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":172,"num_read_kb":105016,"num_write":160,"num_write_kb":46499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"151'79","reported_seq":365,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-12-15T22:58:58.467700+0000","last_change":"2023-12-15T22:58:57.578764+0000","last_active":"2023-12-15T22:58:58.467700+0000","last_peered":"2023-12-15T22:58:58.467700+0000","last_clean":"2023-12-15T22:58:58.467700+0000","last_became_active":"2023-12-15T22:56:39.543179+0000","last_became_peered":"2023-12-15T22:56:39.543179+0000","last_unstale":"2023-12-15T22:58:58.467700+0000","last_undegraded":"2023-12-15T22:58:58.467700+0000","last_fullsized":"2023-12-15T22:58:58.467700+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8369886,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":79,"num_read_kb":71705,"num_write":53,"num_write_kb":14105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":362,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-12-15T22:59:00.466439+0000","last_change":"2023-12-15T22:56:32.607017+0000","last_active":"2023-12-15T22:59:00.466439+0000","last_peered":"2023-12-15T22:59:00.466439+0000","last_clean":"2023-12-15T22:59:00.466439+0000","last_became_active":"2023-12-15T22:56:32.602384+0000","last_became_peered":"2023-12-15T22:56:32.602384+0000","last_unstale":"2023-12-15T22:59:00.466439+0000","last_undegraded":"2023-12-15T22:59:00.466439+0000","last_fullsized":"2023-12-15T22:59:00.466439+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_clean_scrub_stamp":"2023-12-15T22:56:31.525572+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":225,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-12-15T22:58:58.464462+0000","last_change":"2023-12-15T22:56:33.627481+0000","last_active":"2023-12-15T22:58:58.464462+0000","last_peered":"2023-12-15T22:58:58.464462+0000","last_clean":"2023-12-15T22:58:58.464462+0000","last_became_active":"2023-12-15T22:56:33.624689+0000","last_became_peered":"2023-12-15T22:56:33.624689+0000","last_unstale":"2023-12-15T22:58:58.464462+0000","last_undegraded":"2023-12-15T22:58:58.464462+0000","last_fullsized":"2023-12-15T22:58:58.464462+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"151'120","reported_seq":397,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-12-15T22:58:58.459894+0000","last_change":"2023-12-15T22:58:57.521431+0000","last_active":"2023-12-15T22:58:58.459894+0000","last_peered":"2023-12-15T22:58:58.459894+0000","last_clean":"2023-12-15T22:58:58.459894+0000","last_became_active":"2023-12-15T22:56:39.529359+0000","last_became_peered":"2023-12-15T22:56:39.529359+0000","last_unstale":"2023-12-15T22:58:58.459894+0000","last_undegraded":"2023-12-15T22:58:58.459894+0000","last_fullsized":"2023-12-15T22:58:58.459894+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8319498,"num_objects":13,"num_object_clones":7,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":95,"num_read_kb":68055,"num_write":75,"num_write_kb":22851,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":233,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-12-15T22:58:58.542403+0000","last_change":"2023-12-15T22:56:33.655797+0000","last_active":"2023-12-15T22:58:58.542403+0000","last_peered":"2023-12-15T22:58:58.542403+0000","last_clean":"2023-12-15T22:58:58.542403+0000","last_became_active":"2023-12-15T22:56:33.652885+0000","last_became_peered":"2023-12-15T22:56:33.652885+0000","last_unstale":"2023-12-15T22:58:58.542403+0000","last_undegraded":"2023-12-15T22:58:58.542403+0000","last_fullsized":"2023-12-15T22:58:58.542403+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"153'189","reported_seq":421,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-12-15T22:59:00.470098+0000","last_change":"2023-12-15T22:58:59.574184+0000","last_active":"2023-12-15T22:59:00.470098+0000","last_peered":"2023-12-15T22:59:00.470098+0000","last_clean":"2023-12-15T22:59:00.470098+0000","last_became_active":"2023-12-15T22:56:39.543509+0000","last_became_peered":"2023-12-15T22:56:39.543509+0000","last_unstale":"2023-12-15T22:59:00.470098+0000","last_undegraded":"2023-12-15T22:59:00.470098+0000","last_fullsized":"2023-12-15T22:59:00.470098+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8420373,"num_objects":20,"num_object_clones":14,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":110,"num_read_kb":75505,"num_write":85,"num_write_kb":25517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":225,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-12-15T22:58:58.461377+0000","last_change":"2023-12-15T22:56:33.600485+0000","last_active":"2023-12-15T22:58:58.461377+0000","last_peered":"2023-12-15T22:58:58.461377+0000","last_clean":"2023-12-15T22:58:58.461377+0000","last_became_active":"2023-12-15T22:56:33.597320+0000","last_became_peered":"2023-12-15T22:56:33.597320+0000","last_unstale":"2023-12-15T22:58:58.461377+0000","last_undegraded":"2023-12-15T22:58:58.461377+0000","last_fullsized":"2023-12-15T22:58:58.461377+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"153'70","reported_seq":337,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-12-15T22:59:00.467312+0000","last_change":"2023-12-15T22:58:59.535579+0000","last_active":"2023-12-15T22:59:00.467312+0000","last_peered":"2023-12-15T22:59:00.467312+0000","last_clean":"2023-12-15T22:59:00.467312+0000","last_became_active":"2023-12-15T22:56:39.560980+0000","last_became_peered":"2023-12-15T22:56:39.560980+0000","last_unstale":"2023-12-15T22:59:00.467312+0000","last_undegraded":"2023-12-15T22:59:00.467312+0000","last_fullsized":"2023-12-15T22:59:00.467312+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5932904,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":29099,"num_write":30,"num_write_kb":9010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":233,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-12-15T22:58:58.541788+0000","last_change":"2023-12-15T22:56:33.648541+0000","last_active":"2023-12-15T22:58:58.541788+0000","last_peered":"2023-12-15T22:58:58.541788+0000","last_clean":"2023-12-15T22:58:58.541788+0000","last_became_active":"2023-12-15T22:56:33.643681+0000","last_became_peered":"2023-12-15T22:56:33.643681+0000","last_unstale":"2023-12-15T22:58:58.541788+0000","last_undegraded":"2023-12-15T22:58:58.541788+0000","last_fullsized":"2023-12-15T22:58:58.541788+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"152'128","reported_seq":375,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-12-15T22:58:58.560460+0000","last_change":"2023-12-15T22:58:57.563578+0000","last_active":"2023-12-15T22:58:58.560460+0000","last_peered":"2023-12-15T22:58:58.560460+0000","last_clean":"2023-12-15T22:58:58.560460+0000","last_became_active":"2023-12-15T22:56:39.506510+0000","last_became_peered":"2023-12-15T22:56:39.506510+0000","last_unstale":"2023-12-15T22:58:58.560460+0000","last_undegraded":"2023-12-15T22:58:58.560460+0000","last_fullsized":"2023-12-15T22:58:58.560460+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8638864,"num_objects":15,"num_object_clones":10,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":80,"num_read_kb":55526,"num_write":66,"num_write_kb":19376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"151'66","reported_seq":316,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-12-15T22:58:58.462633+0000","last_change":"2023-12-15T22:58:57.535680+0000","last_active":"2023-12-15T22:58:58.462633+0000","last_peered":"2023-12-15T22:58:58.462633+0000","last_clean":"2023-12-15T22:58:58.462633+0000","last_became_active":"2023-12-15T22:56:39.511159+0000","last_became_peered":"2023-12-15T22:56:39.511159+0000","last_unstale":"2023-12-15T22:58:58.462633+0000","last_undegraded":"2023-12-15T22:58:58.462633+0000","last_fullsized":"2023-12-15T22:58:58.462633+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2403434,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":12,"num_read_kb":7,"num_write":33,"num_write_kb":9807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":219,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-12-15T22:58:58.473986+0000","last_change":"2023-12-15T22:56:33.633890+0000","last_active":"2023-12-15T22:58:58.473986+0000","last_peered":"2023-12-15T22:58:58.473986+0000","last_clean":"2023-12-15T22:58:58.473986+0000","last_became_active":"2023-12-15T22:56:33.631134+0000","last_became_peered":"2023-12-15T22:56:33.631134+0000","last_unstale":"2023-12-15T22:58:58.473986+0000","last_undegraded":"2023-12-15T22:58:58.473986+0000","last_fullsized":"2023-12-15T22:58:58.473986+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"150'75","reported_seq":329,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-12-15T22:58:58.470048+0000","last_change":"2023-12-15T22:58:57.572665+0000","last_active":"2023-12-15T22:58:58.470048+0000","last_peered":"2023-12-15T22:58:58.470048+0000","last_clean":"2023-12-15T22:58:58.470048+0000","last_became_active":"2023-12-15T22:56:39.532730+0000","last_became_peered":"2023-12-15T22:56:39.532730+0000","last_unstale":"2023-12-15T22:58:58.470048+0000","last_undegraded":"2023-12-15T22:58:58.470048+0000","last_fullsized":"2023-12-15T22:58:58.470048+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2437048,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":45,"num_read_kb":38480,"num_write":35,"num_write_kb":10622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":233,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-12-15T22:59:00.464165+0000","last_change":"2023-12-15T22:56:33.655736+0000","last_active":"2023-12-15T22:59:00.464165+0000","last_peered":"2023-12-15T22:59:00.464165+0000","last_clean":"2023-12-15T22:59:00.464165+0000","last_became_active":"2023-12-15T22:56:33.652917+0000","last_became_peered":"2023-12-15T22:56:33.652917+0000","last_unstale":"2023-12-15T22:59:00.464165+0000","last_undegraded":"2023-12-15T22:59:00.464165+0000","last_fullsized":"2023-12-15T22:59:00.464165+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"153'122","reported_seq":392,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-12-15T22:59:00.469717+0000","last_change":"2023-12-15T22:58:59.538125+0000","last_active":"2023-12-15T22:59:00.469717+0000","last_peered":"2023-12-15T22:59:00.469717+0000","last_clean":"2023-12-15T22:59:00.469717+0000","last_became_active":"2023-12-15T22:56:39.557252+0000","last_became_peered":"2023-12-15T22:56:39.557252+0000","last_unstale":"2023-12-15T22:59:00.469717+0000","last_undegraded":"2023-12-15T22:59:00.469717+0000","last_fullsized":"2023-12-15T22:59:00.469717+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8269110,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":74,"num_read_kb":54273,"num_write":68,"num_write_kb":19580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":219,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-12-15T22:58:58.470491+0000","last_change":"2023-12-15T22:56:33.632348+0000","last_active":"2023-12-15T22:58:58.470491+0000","last_peered":"2023-12-15T22:58:58.470491+0000","last_clean":"2023-12-15T22:58:58.470491+0000","last_became_active":"2023-12-15T22:56:33.629022+0000","last_became_peered":"2023-12-15T22:56:33.629022+0000","last_unstale":"2023-12-15T22:58:58.470491+0000","last_undegraded":"2023-12-15T22:58:58.470491+0000","last_fullsized":"2023-12-15T22:58:58.470491+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"150'42","reported_seq":307,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-12-15T22:58:58.478478+0000","last_change":"2023-12-15T22:58:57.550694+0000","last_active":"2023-12-15T22:58:58.478478+0000","last_peered":"2023-12-15T22:58:58.478478+0000","last_clean":"2023-12-15T22:58:58.478478+0000","last_became_active":"2023-12-15T22:56:39.535145+0000","last_became_peered":"2023-12-15T22:56:39.535145+0000","last_unstale":"2023-12-15T22:58:58.478478+0000","last_undegraded":"2023-12-15T22:58:58.478478+0000","last_fullsized":"2023-12-15T22:58:58.478478+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2100908,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":29,"num_read_kb":29390,"num_write":17,"num_write_kb":5075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":225,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-12-15T22:58:58.458010+0000","last_change":"2023-12-15T22:56:33.627005+0000","last_active":"2023-12-15T22:58:58.458010+0000","last_peered":"2023-12-15T22:58:58.458010+0000","last_clean":"2023-12-15T22:58:58.458010+0000","last_became_active":"2023-12-15T22:56:33.622181+0000","last_became_peered":"2023-12-15T22:56:33.622181+0000","last_unstale":"2023-12-15T22:58:58.458010+0000","last_undegraded":"2023-12-15T22:58:58.458010+0000","last_fullsized":"2023-12-15T22:58:58.458010+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":107044344,"num_objects":193,"num_object_clones":131,"num_object_copies":579,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":24,"num_read":1139,"num_read_kb":831076,"num_write":940,"num_write_kb":272934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1790,"ondisk_log_size":1790,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":128700,"kb_used_data":97716,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94243140,"statfs":{"total":96636764160,"available":96504975360,"internally_reserved":0,"allocated":100061184,"data_stored":113676504,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":111280,"kb_used_data":80296,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94260560,"statfs":{"total":96636764160,"available":96522813440,"internally_reserved":0,"allocated":82223104,"data_stored":93985520,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":134920,"kb_used_data":103936,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94236920,"statfs":{"total":96636764160,"available":96498606080,"internally_reserved":0,"allocated":106430464,"data_stored":120051966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":131144,"kb_used_data":100160,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94240696,"statfs":{"total":96636764160,"available":96502472704,"internally_reserved":0,"allocated":102563840,"data_stored":113485913,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-15T22:59:02.572 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:59:02.675 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:02.674+0000 7fc382b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:02.678 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:02.679+0000 7fc382b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:03.264 INFO:tasks.rados.rados.0.smithi071.stdout:571: done (1 left) 2023-12-15T22:59:03.265 INFO:tasks.rados.rados.0.smithi071.stdout:572: delete oid 37 current snap is 74 2023-12-15T22:59:03.265 INFO:tasks.rados.rados.0.smithi071.stdout:570: finishing write tid 1 to smithi07183189-46 2023-12-15T22:59:03.265 INFO:tasks.rados.rados.0.smithi071.stdout:570: finishing write tid 2 to smithi07183189-46 2023-12-15T22:59:03.265 INFO:tasks.rados.rados.0.smithi071.stdout:570: finishing write tid 3 to smithi07183189-46 2023-12-15T22:59:03.265 INFO:tasks.rados.rados.0.smithi071.stdout:570: finishing write tid 4 to smithi07183189-46 2023-12-15T22:59:03.265 INFO:tasks.rados.rados.0.smithi071.stdout:570: oid 46 updating version 0 to 81 2023-12-15T22:59:03.265 INFO:tasks.rados.rados.0.smithi071.stdout:570: oid 46 version 81 is already newer than 80 2023-12-15T22:59:03.265 INFO:tasks.rados.rados.0.smithi071.stdout:570: oid 46 updating version 81 to 82 2023-12-15T22:59:03.265 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 82 (ObjNum 277 snap 74 seq_num 277) dirty exists 2023-12-15T22:59:03.265 INFO:tasks.rados.rados.0.smithi071.stdout:570: left oid 46 (ObjNum 277 snap 74 seq_num 277) 2023-12-15T22:59:03.296 INFO:tasks.rados.rados.0.smithi071.stdout:570: done (1 left) 2023-12-15T22:59:03.297 INFO:tasks.rados.rados.0.smithi071.stdout:572: done (0 left) 2023-12-15T22:59:03.297 INFO:tasks.rados.rados.0.smithi071.stdout:573: delete oid 48 current snap is 74 2023-12-15T22:59:03.398 INFO:tasks.rados.rados.0.smithi071.stdout:573: done (0 left) 2023-12-15T22:59:03.398 INFO:tasks.rados.rados.0.smithi071.stdout:574: read oid 9 snap -1 2023-12-15T22:59:03.398 INFO:tasks.rados.rados.0.smithi071.stdout:574: expect (ObjNum 271 snap 70 seq_num 271) 2023-12-15T22:59:03.398 INFO:tasks.rados.rados.0.smithi071.stdout:575: snap_create 2023-12-15T22:59:03.410 INFO:tasks.rados.rados.0.smithi071.stdout:574: expect (ObjNum 271 snap 70 seq_num 271) 2023-12-15T22:59:04.265 INFO:tasks.rados.rados.0.smithi071.stdout:574: done (1 left) 2023-12-15T22:59:04.265 INFO:tasks.rados.rados.0.smithi071.stdout:575: done (0 left) 2023-12-15T22:59:04.265 INFO:tasks.rados.rados.0.smithi071.stdout:576: write oid 45 current snap is 75 2023-12-15T22:59:04.265 INFO:tasks.rados.rados.0.smithi071.stdout:576: seq_num 280 ranges {705960=458428,1735311=717424} 2023-12-15T22:59:04.271 INFO:tasks.rados.rados.0.smithi071.stdout:576: writing smithi07183189-45 from 705960 to 1164388 tid 1 2023-12-15T22:59:04.279 INFO:tasks.rados.rados.0.smithi071.stdout:576: writing smithi07183189-45 from 1735311 to 2452735 tid 2 2023-12-15T22:59:04.279 INFO:tasks.rados.rados.0.smithi071.stdout:577: read oid 2 snap 58 2023-12-15T22:59:04.279 INFO:tasks.rados.rados.0.smithi071.stdout:577: expect deleted 2023-12-15T22:59:04.279 INFO:tasks.rados.rados.0.smithi071.stdout:578: snap_remove snap 70 2023-12-15T22:59:04.749 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:59:04.852 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:04.850+0000 7f3f09f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:04.855 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:04.855+0000 7f3f09f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:05.271 INFO:tasks.rados.rados.0.smithi071.stdout:578: done (2 left) 2023-12-15T22:59:05.271 INFO:tasks.rados.rados.0.smithi071.stdout:579: snap_remove snap 74 2023-12-15T22:59:06.273 INFO:tasks.rados.rados.0.smithi071.stdout:579: done (2 left) 2023-12-15T22:59:06.274 INFO:tasks.rados.rados.0.smithi071.stdout:580: write (excl) oid 40 current snap is 75 2023-12-15T22:59:06.274 INFO:tasks.rados.rados.0.smithi071.stdout:580: seq_num 281 ranges {722767=533677,1793793=461082} 2023-12-15T22:59:06.281 INFO:tasks.rados.rados.0.smithi071.stdout:580: writing smithi07183189-40 from 722767 to 1256444 tid 1 2023-12-15T22:59:06.289 INFO:tasks.rados.rados.0.smithi071.stdout:580: writing smithi07183189-40 from 1793793 to 2254875 tid 2 2023-12-15T22:59:06.289 INFO:tasks.rados.rados.0.smithi071.stdout:581: read oid 50 snap -1 2023-12-15T22:59:06.289 INFO:tasks.rados.rados.0.smithi071.stdout:581: expect (ObjNum 227 snap 55 seq_num 227) 2023-12-15T22:59:06.289 INFO:tasks.rados.rados.0.smithi071.stdout:582: write oid 19 current snap is 75 2023-12-15T22:59:06.289 INFO:tasks.rados.rados.0.smithi071.stdout:582: seq_num 282 ranges {739574=725279,1968628=721093} 2023-12-15T22:59:06.298 INFO:tasks.rados.rados.0.smithi071.stdout:582: writing smithi07183189-19 from 739574 to 1464853 tid 1 2023-12-15T22:59:06.308 INFO:tasks.rados.rados.0.smithi071.stdout:582: writing smithi07183189-19 from 1968628 to 2689721 tid 2 2023-12-15T22:59:06.308 INFO:tasks.rados.rados.0.smithi071.stdout:583: snap_create 2023-12-15T22:59:06.308 INFO:tasks.rados.rados.0.smithi071.stdout:576: finishing write tid 1 to smithi07183189-45 2023-12-15T22:59:06.308 INFO:tasks.rados.rados.0.smithi071.stdout:576: finishing write tid 2 to smithi07183189-45 2023-12-15T22:59:06.309 INFO:tasks.rados.rados.0.smithi071.stdout:576: finishing write tid 3 to smithi07183189-45 2023-12-15T22:59:06.309 INFO:tasks.rados.rados.0.smithi071.stdout:576: finishing write tid 4 to smithi07183189-45 2023-12-15T22:59:06.309 INFO:tasks.rados.rados.0.smithi071.stdout:576: oid 45 updating version 0 to 359 2023-12-15T22:59:06.309 INFO:tasks.rados.rados.0.smithi071.stdout:576: oid 45 updating version 359 to 361 2023-12-15T22:59:06.309 INFO:tasks.rados.rados.0.smithi071.stdout:576: oid 45 version 361 is already newer than 360 2023-12-15T22:59:06.309 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 361 (ObjNum 280 snap 75 seq_num 280) dirty exists 2023-12-15T22:59:06.309 INFO:tasks.rados.rados.0.smithi071.stdout:576: left oid 45 (ObjNum 280 snap 75 seq_num 280) 2023-12-15T22:59:06.372 INFO:tasks.rados.rados.0.smithi071.stdout:580: finishing write tid 1 to smithi07183189-40 2023-12-15T22:59:06.373 INFO:tasks.rados.rados.0.smithi071.stdout:582: finishing write tid 1 to smithi07183189-19 2023-12-15T22:59:06.394 INFO:tasks.rados.rados.0.smithi071.stdout:580: finishing write tid 2 to smithi07183189-40 2023-12-15T22:59:06.413 INFO:tasks.rados.rados.0.smithi071.stdout:582: finishing write tid 2 to smithi07183189-19 2023-12-15T22:59:06.414 INFO:tasks.rados.rados.0.smithi071.stdout:580: finishing write tid 3 to smithi07183189-40 2023-12-15T22:59:06.416 INFO:tasks.rados.rados.0.smithi071.stdout:582: finishing write tid 3 to smithi07183189-19 2023-12-15T22:59:06.418 INFO:tasks.rados.rados.0.smithi071.stdout:580: finishing write tid 4 to smithi07183189-40 2023-12-15T22:59:06.418 INFO:tasks.rados.rados.0.smithi071.stdout:580: oid 40 updating version 0 to 369 2023-12-15T22:59:06.418 INFO:tasks.rados.rados.0.smithi071.stdout:580: oid 40 updating version 369 to 371 2023-12-15T22:59:06.418 INFO:tasks.rados.rados.0.smithi071.stdout:580: oid 40 version 371 is already newer than 370 2023-12-15T22:59:06.418 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 40 v 371 (ObjNum 281 snap 75 seq_num 281) dirty exists 2023-12-15T22:59:06.418 INFO:tasks.rados.rados.0.smithi071.stdout:580: left oid 40 (ObjNum 281 snap 75 seq_num 281) 2023-12-15T22:59:06.419 INFO:tasks.rados.rados.0.smithi071.stdout:582: finishing write tid 4 to smithi07183189-19 2023-12-15T22:59:06.419 INFO:tasks.rados.rados.0.smithi071.stdout:582: oid 19 updating version 0 to 163 2023-12-15T22:59:06.419 INFO:tasks.rados.rados.0.smithi071.stdout:582: oid 19 version 163 is already newer than 161 2023-12-15T22:59:06.419 INFO:tasks.rados.rados.0.smithi071.stdout:582: oid 19 version 163 is already newer than 163 2023-12-15T22:59:06.419 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 163 (ObjNum 282 snap 75 seq_num 282) dirty exists 2023-12-15T22:59:06.419 INFO:tasks.rados.rados.0.smithi071.stdout:582: left oid 19 (ObjNum 282 snap 75 seq_num 282) 2023-12-15T22:59:06.429 INFO:tasks.rados.rados.0.smithi071.stdout:581: expect (ObjNum 227 snap 55 seq_num 227) 2023-12-15T22:59:06.923 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:59:07.027 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:07.026+0000 7fca6082d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:07.030 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:07.030+0000 7fca6082d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:07.283 INFO:tasks.rados.rados.0.smithi071.stdout:576: done (5 left) 2023-12-15T22:59:07.283 INFO:tasks.rados.rados.0.smithi071.stdout:577: done (4 left) 2023-12-15T22:59:07.283 INFO:tasks.rados.rados.0.smithi071.stdout:580: done (3 left) 2023-12-15T22:59:07.283 INFO:tasks.rados.rados.0.smithi071.stdout:581: done (2 left) 2023-12-15T22:59:07.283 INFO:tasks.rados.rados.0.smithi071.stdout:582: done (1 left) 2023-12-15T22:59:07.283 INFO:tasks.rados.rados.0.smithi071.stdout:583: done (0 left) 2023-12-15T22:59:07.283 INFO:tasks.rados.rados.0.smithi071.stdout:584: write (excl) oid 28 current snap is 76 2023-12-15T22:59:07.283 INFO:tasks.rados.rados.0.smithi071.stdout:584: seq_num 283 ranges {756381=400528,1910757=464751} 2023-12-15T22:59:07.290 INFO:tasks.rados.rados.0.smithi071.stdout:584: writing smithi07183189-28 from 756381 to 1156909 tid 1 2023-12-15T22:59:07.298 INFO:tasks.rados.rados.0.smithi071.stdout:584: writing smithi07183189-28 from 1910757 to 2375508 tid 2 2023-12-15T22:59:07.298 INFO:tasks.rados.rados.0.smithi071.stdout:585: snap_create 2023-12-15T22:59:07.345 INFO:tasks.rados.rados.0.smithi071.stdout:584: finishing write tid 1 to smithi07183189-28 2023-12-15T22:59:07.364 INFO:tasks.rados.rados.0.smithi071.stdout:584: finishing write tid 2 to smithi07183189-28 2023-12-15T22:59:07.373 INFO:tasks.rados.rados.0.smithi071.stdout:584: finishing write tid 3 to smithi07183189-28 2023-12-15T22:59:07.374 INFO:tasks.rados.rados.0.smithi071.stdout:584: finishing write tid 4 to smithi07183189-28 2023-12-15T22:59:07.374 INFO:tasks.rados.rados.0.smithi071.stdout:584: oid 28 updating version 0 to 95 2023-12-15T22:59:07.375 INFO:tasks.rados.rados.0.smithi071.stdout:584: oid 28 version 95 is already newer than 94 2023-12-15T22:59:07.375 INFO:tasks.rados.rados.0.smithi071.stdout:584: oid 28 version 95 is already newer than 93 2023-12-15T22:59:07.375 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 95 (ObjNum 283 snap 76 seq_num 283) dirty exists 2023-12-15T22:59:07.375 INFO:tasks.rados.rados.0.smithi071.stdout:584: left oid 28 (ObjNum 283 snap 76 seq_num 283) 2023-12-15T22:59:07.451 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T22:59:07.451 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T22:59:07.452 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-12-15T22:59:07.452 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-12-15T22:59:07.558 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:07.553+0000 7f5d7ed9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:07.561 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:07.561+0000 7f5d7ed9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:07.575 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:59:07.575 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:59:07.575 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:59:07.575 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:59:07.575 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:59:07.575 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:59:07.575 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:59:07.575 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:59:07.575 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:59:07.575 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:59:07.575 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:59:07.575 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:59:07.579 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:59:07.579 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:59:07.579 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-12-15T22:59:07.684 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:07.679+0000 7f4973d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:07.687 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:07.687+0000 7f4973d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:07.698 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:59:07.699 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:59:07.699 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:59:07.699 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:59:07.699 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:59:07.699 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:59:07.699 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:59:07.699 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:59:07.699 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:59:07.699 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:59:07.699 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:59:07.699 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:59:07.701 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:59:07.702 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:59:07.702 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-12-15T22:59:07.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:07.803+0000 7fc98139e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:07.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:07.810+0000 7fc98139e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:07.823 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:59:07.824 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:59:07.824 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:59:07.824 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:59:07.824 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:59:07.824 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:59:07.824 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:59:07.824 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:59:07.824 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:59:07.824 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:59:07.824 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:59:07.824 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:59:07.826 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:59:07.826 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:59:07.827 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-12-15T22:59:07.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:07.928+0000 7f988e06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:07.936 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:07.936+0000 7f988e06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:07.949 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:59:07.949 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:59:07.949 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:59:07.949 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:59:07.949 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:59:07.949 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:59:07.949 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:59:07.949 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:59:07.949 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:59:07.949 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:59:07.949 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:59:07.949 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:59:07.952 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:59:07.952 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:59:08.287 INFO:tasks.rados.rados.0.smithi071.stdout:584: done (1 left) 2023-12-15T22:59:08.287 INFO:tasks.rados.rados.0.smithi071.stdout:585: done (0 left) 2023-12-15T22:59:08.287 INFO:tasks.rados.rados.0.smithi071.stdout:586: write (excl) oid 19 current snap is 77 2023-12-15T22:59:08.287 INFO:tasks.rados.rados.0.smithi071.stdout:586: seq_num 284 ranges {773188=475777,1969239=724762} 2023-12-15T22:59:08.294 INFO:tasks.rados.rados.0.smithi071.stdout:586: writing smithi07183189-19 from 773188 to 1248965 tid 1 2023-12-15T22:59:08.304 INFO:tasks.rados.rados.0.smithi071.stdout:586: writing smithi07183189-19 from 1969239 to 2694001 tid 2 2023-12-15T22:59:08.304 INFO:tasks.rados.rados.0.smithi071.stdout:587: snap_remove snap 72 2023-12-15T22:59:09.100 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:59:09.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:09.216+0000 7f5af6d04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:09.220 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:09.220+0000 7f5af6d04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:09.296 INFO:tasks.rados.rados.0.smithi071.stdout:587: done (1 left) 2023-12-15T22:59:09.296 INFO:tasks.rados.rados.0.smithi071.stdout:588: write (excl) oid 50 current snap is 77 2023-12-15T22:59:09.296 INFO:tasks.rados.rados.0.smithi071.stdout:586: finishing write tid 1 to smithi07183189-19 2023-12-15T22:59:09.296 INFO:tasks.rados.rados.0.smithi071.stdout:588: seq_num 285 ranges {789995=551026,2027721=468420} 2023-12-15T22:59:09.303 INFO:tasks.rados.rados.0.smithi071.stdout:588: writing smithi07183189-50 from 789995 to 1341021 tid 1 2023-12-15T22:59:09.310 INFO:tasks.rados.rados.0.smithi071.stdout:588: writing smithi07183189-50 from 2027721 to 2496141 tid 2 2023-12-15T22:59:09.310 INFO:tasks.rados.rados.0.smithi071.stdout:589: snap_create 2023-12-15T22:59:09.310 INFO:tasks.rados.rados.0.smithi071.stdout:586: finishing write tid 2 to smithi07183189-19 2023-12-15T22:59:09.310 INFO:tasks.rados.rados.0.smithi071.stdout:586: finishing write tid 3 to smithi07183189-19 2023-12-15T22:59:09.310 INFO:tasks.rados.rados.0.smithi071.stdout:586: finishing write tid 4 to smithi07183189-19 2023-12-15T22:59:09.311 INFO:tasks.rados.rados.0.smithi071.stdout:586: oid 19 updating version 0 to 165 2023-12-15T22:59:09.311 INFO:tasks.rados.rados.0.smithi071.stdout:586: oid 19 updating version 165 to 166 2023-12-15T22:59:09.311 INFO:tasks.rados.rados.0.smithi071.stdout:586: oid 19 updating version 166 to 167 2023-12-15T22:59:09.311 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 167 (ObjNum 284 snap 77 seq_num 284) dirty exists 2023-12-15T22:59:09.311 INFO:tasks.rados.rados.0.smithi071.stdout:586: left oid 19 (ObjNum 284 snap 77 seq_num 284) 2023-12-15T22:59:09.372 INFO:tasks.rados.rados.0.smithi071.stdout:588: finishing write tid 1 to smithi07183189-50 2023-12-15T22:59:09.392 INFO:tasks.rados.rados.0.smithi071.stdout:588: finishing write tid 2 to smithi07183189-50 2023-12-15T22:59:09.393 INFO:tasks.rados.rados.0.smithi071.stdout:588: finishing write tid 3 to smithi07183189-50 2023-12-15T22:59:09.394 INFO:tasks.rados.rados.0.smithi071.stdout:588: finishing write tid 4 to smithi07183189-50 2023-12-15T22:59:09.395 INFO:tasks.rados.rados.0.smithi071.stdout:588: oid 50 updating version 0 to 377 2023-12-15T22:59:09.395 INFO:tasks.rados.rados.0.smithi071.stdout:588: oid 50 version 377 is already newer than 375 2023-12-15T22:59:09.395 INFO:tasks.rados.rados.0.smithi071.stdout:588: oid 50 version 377 is already newer than 377 2023-12-15T22:59:09.395 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 377 (ObjNum 285 snap 77 seq_num 285) dirty exists 2023-12-15T22:59:09.395 INFO:tasks.rados.rados.0.smithi071.stdout:588: left oid 50 (ObjNum 285 snap 77 seq_num 285) 2023-12-15T22:59:10.295 INFO:tasks.rados.rados.0.smithi071.stdout:586: done (2 left) 2023-12-15T22:59:10.295 INFO:tasks.rados.rados.0.smithi071.stdout:588: done (1 left) 2023-12-15T22:59:10.295 INFO:tasks.rados.rados.0.smithi071.stdout:589: done (0 left) 2023-12-15T22:59:10.295 INFO:tasks.rados.rados.0.smithi071.stdout:590: write (excl) oid 2 current snap is 78 2023-12-15T22:59:10.295 INFO:tasks.rados.rados.0.smithi071.stdout:590: seq_num 286 ranges {406802=626275,1686203=728431} 2023-12-15T22:59:10.301 INFO:tasks.rados.rados.0.smithi071.stdout:590: writing smithi07183189-2 from 406802 to 1033077 tid 1 2023-12-15T22:59:10.309 INFO:tasks.rados.rados.0.smithi071.stdout:590: writing smithi07183189-2 from 1686203 to 2414634 tid 2 2023-12-15T22:59:10.309 INFO:tasks.rados.rados.0.smithi071.stdout:591: read oid 35 snap -1 2023-12-15T22:59:10.309 INFO:tasks.rados.rados.0.smithi071.stdout:591: expect (ObjNum 272 snap 70 seq_num 272) 2023-12-15T22:59:10.309 INFO:tasks.rados.rados.0.smithi071.stdout:592: delete oid 8 current snap is 78 2023-12-15T22:59:10.357 INFO:tasks.rados.rados.0.smithi071.stdout:592: done (2 left) 2023-12-15T22:59:10.358 INFO:tasks.rados.rados.0.smithi071.stdout:593: write (excl) oid 32 current snap is 78 2023-12-15T22:59:10.358 INFO:tasks.rados.rados.0.smithi071.stdout:593: seq_num 288 ranges {440416=776773,1686814=732100} 2023-12-15T22:59:10.365 INFO:tasks.rados.rados.0.smithi071.stdout:593: writing smithi07183189-32 from 440416 to 1217189 tid 1 2023-12-15T22:59:10.372 INFO:tasks.rados.rados.0.smithi071.stdout:593: writing smithi07183189-32 from 1686814 to 2418914 tid 2 2023-12-15T22:59:10.372 INFO:tasks.rados.rados.0.smithi071.stdout:594: read oid 26 snap -1 2023-12-15T22:59:10.372 INFO:tasks.rados.rados.0.smithi071.stdout:594: expect (ObjNum 259 snap 70 seq_num 259) 2023-12-15T22:59:10.372 INFO:tasks.rados.rados.0.smithi071.stdout:595: write oid 30 current snap is 78 2023-12-15T22:59:10.372 INFO:tasks.rados.rados.0.smithi071.stdout:595: seq_num 289 ranges {457223=568375,1461649=475758,2420186=437037} 2023-12-15T22:59:10.378 INFO:tasks.rados.rados.0.smithi071.stdout:595: writing smithi07183189-30 from 457223 to 1025598 tid 1 2023-12-15T22:59:10.384 INFO:tasks.rados.rados.0.smithi071.stdout:595: writing smithi07183189-30 from 1461649 to 1937407 tid 2 2023-12-15T22:59:10.389 INFO:tasks.rados.rados.0.smithi071.stdout:595: writing smithi07183189-30 from 2420186 to 2857223 tid 3 2023-12-15T22:59:10.389 INFO:tasks.rados.rados.0.smithi071.stdout:596: write (excl) oid 5 current snap is 78 2023-12-15T22:59:10.389 INFO:tasks.rados.rados.0.smithi071.stdout:596: seq_num 290 ranges {474030=643624,1520131=735769} 2023-12-15T22:59:10.395 INFO:tasks.rados.rados.0.smithi071.stdout:596: writing smithi07183189-5 from 474030 to 1117654 tid 1 2023-12-15T22:59:10.402 INFO:tasks.rados.rados.0.smithi071.stdout:596: writing smithi07183189-5 from 1520131 to 2255900 tid 2 2023-12-15T22:59:10.402 INFO:tasks.rados.rados.0.smithi071.stdout:597: write (excl) oid 6 current snap is 78 2023-12-15T22:59:10.402 INFO:tasks.rados.rados.0.smithi071.stdout:597: seq_num 291 ranges {490837=718873,1862260=479427,2875032=15805} 2023-12-15T22:59:10.409 INFO:tasks.rados.rados.0.smithi071.stdout:597: writing smithi07183189-6 from 490837 to 1209710 tid 1 2023-12-15T22:59:10.416 INFO:tasks.rados.rados.0.smithi071.stdout:597: writing smithi07183189-6 from 1862260 to 2341687 tid 2 2023-12-15T22:59:10.419 INFO:tasks.rados.rados.0.smithi071.stdout:597: writing smithi07183189-6 from 2875032 to 2890837 tid 3 2023-12-15T22:59:10.419 INFO:tasks.rados.rados.0.smithi071.stdout:598: read oid 39 snap 76 2023-12-15T22:59:10.419 INFO:tasks.rados.rados.0.smithi071.stdout:598: expect deleted 2023-12-15T22:59:10.420 INFO:tasks.rados.rados.0.smithi071.stdout:599: delete oid 21 current snap is 78 2023-12-15T22:59:10.420 INFO:tasks.rados.rados.0.smithi071.stdout:594: expect (ObjNum 259 snap 70 seq_num 259) 2023-12-15T22:59:10.571 INFO:tasks.rados.rados.0.smithi071.stdout:591: expect (ObjNum 272 snap 70 seq_num 272) 2023-12-15T22:59:10.769 INFO:tasks.rados.rados.0.smithi071.stdout:590: finishing write tid 1 to smithi07183189-2 2023-12-15T22:59:10.769 INFO:tasks.rados.rados.0.smithi071.stdout:595: finishing write tid 1 to smithi07183189-30 2023-12-15T22:59:10.769 INFO:tasks.rados.rados.0.smithi071.stdout:593: finishing write tid 1 to smithi07183189-32 2023-12-15T22:59:10.769 INFO:tasks.rados.rados.0.smithi071.stdout:590: finishing write tid 2 to smithi07183189-2 2023-12-15T22:59:10.769 INFO:tasks.rados.rados.0.smithi071.stdout:595: finishing write tid 2 to smithi07183189-30 2023-12-15T22:59:10.769 INFO:tasks.rados.rados.0.smithi071.stdout:591: done (8 left) 2023-12-15T22:59:10.769 INFO:tasks.rados.rados.0.smithi071.stdout:594: done (7 left) 2023-12-15T22:59:10.769 INFO:tasks.rados.rados.0.smithi071.stdout:598: done (6 left) 2023-12-15T22:59:10.769 INFO:tasks.rados.rados.0.smithi071.stdout:599: done (5 left) 2023-12-15T22:59:10.769 INFO:tasks.rados.rados.0.smithi071.stdout:600: snap_remove snap 71 2023-12-15T22:59:11.292 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:59:11.307 INFO:tasks.rados.rados.0.smithi071.stdout:600: done (5 left) 2023-12-15T22:59:11.307 INFO:tasks.rados.rados.0.smithi071.stdout:601: snap_create 2023-12-15T22:59:11.307 INFO:tasks.rados.rados.0.smithi071.stdout:596: finishing write tid 1 to smithi07183189-5 2023-12-15T22:59:11.308 INFO:tasks.rados.rados.0.smithi071.stdout:593: finishing write tid 2 to smithi07183189-32 2023-12-15T22:59:11.308 INFO:tasks.rados.rados.0.smithi071.stdout:597: finishing write tid 1 to smithi07183189-6 2023-12-15T22:59:11.308 INFO:tasks.rados.rados.0.smithi071.stdout:590: finishing write tid 3 to smithi07183189-2 2023-12-15T22:59:11.308 INFO:tasks.rados.rados.0.smithi071.stdout:590: finishing write tid 4 to smithi07183189-2 2023-12-15T22:59:11.308 INFO:tasks.rados.rados.0.smithi071.stdout:590: oid 2 updating version 0 to 100 2023-12-15T22:59:11.308 INFO:tasks.rados.rados.0.smithi071.stdout:590: oid 2 version 100 is already newer than 99 2023-12-15T22:59:11.308 INFO:tasks.rados.rados.0.smithi071.stdout:590: oid 2 version 100 is already newer than 98 2023-12-15T22:59:11.308 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 100 (ObjNum 286 snap 78 seq_num 286) dirty exists 2023-12-15T22:59:11.308 INFO:tasks.rados.rados.0.smithi071.stdout:590: left oid 2 (ObjNum 286 snap 78 seq_num 286) 2023-12-15T22:59:11.308 INFO:tasks.rados.rados.0.smithi071.stdout:596: finishing write tid 2 to smithi07183189-5 2023-12-15T22:59:11.308 INFO:tasks.rados.rados.0.smithi071.stdout:593: finishing write tid 3 to smithi07183189-32 2023-12-15T22:59:11.308 INFO:tasks.rados.rados.0.smithi071.stdout:595: finishing write tid 3 to smithi07183189-30 2023-12-15T22:59:11.308 INFO:tasks.rados.rados.0.smithi071.stdout:593: finishing write tid 4 to smithi07183189-32 2023-12-15T22:59:11.308 INFO:tasks.rados.rados.0.smithi071.stdout:593: oid 32 updating version 0 to 28 2023-12-15T22:59:11.308 INFO:tasks.rados.rados.0.smithi071.stdout:593: oid 32 version 28 is already newer than 27 2023-12-15T22:59:11.308 INFO:tasks.rados.rados.0.smithi071.stdout:593: oid 32 version 28 is already newer than 26 2023-12-15T22:59:11.309 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 28 (ObjNum 288 snap 78 seq_num 288) dirty exists 2023-12-15T22:59:11.309 INFO:tasks.rados.rados.0.smithi071.stdout:593: left oid 32 (ObjNum 288 snap 78 seq_num 288) 2023-12-15T22:59:11.309 INFO:tasks.rados.rados.0.smithi071.stdout:597: finishing write tid 2 to smithi07183189-6 2023-12-15T22:59:11.309 INFO:tasks.rados.rados.0.smithi071.stdout:596: finishing write tid 3 to smithi07183189-5 2023-12-15T22:59:11.309 INFO:tasks.rados.rados.0.smithi071.stdout:595: finishing write tid 4 to smithi07183189-30 2023-12-15T22:59:11.309 INFO:tasks.rados.rados.0.smithi071.stdout:596: finishing write tid 4 to smithi07183189-5 2023-12-15T22:59:11.309 INFO:tasks.rados.rados.0.smithi071.stdout:596: oid 5 updating version 0 to 381 2023-12-15T22:59:11.309 INFO:tasks.rados.rados.0.smithi071.stdout:596: oid 5 version 381 is already newer than 380 2023-12-15T22:59:11.309 INFO:tasks.rados.rados.0.smithi071.stdout:596: oid 5 version 381 is already newer than 379 2023-12-15T22:59:11.309 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 5 v 381 (ObjNum 290 snap 78 seq_num 290) dirty exists 2023-12-15T22:59:11.309 INFO:tasks.rados.rados.0.smithi071.stdout:596: left oid 5 (ObjNum 290 snap 78 seq_num 290) 2023-12-15T22:59:11.309 INFO:tasks.rados.rados.0.smithi071.stdout:595: finishing write tid 5 to smithi07183189-30 2023-12-15T22:59:11.309 INFO:tasks.rados.rados.0.smithi071.stdout:595: oid 30 updating version 0 to 74 2023-12-15T22:59:11.309 INFO:tasks.rados.rados.0.smithi071.stdout:595: oid 30 updating version 74 to 77 2023-12-15T22:59:11.310 INFO:tasks.rados.rados.0.smithi071.stdout:595: oid 30 version 77 is already newer than 75 2023-12-15T22:59:11.310 INFO:tasks.rados.rados.0.smithi071.stdout:595: oid 30 version 77 is already newer than 76 2023-12-15T22:59:11.310 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 77 (ObjNum 289 snap 78 seq_num 289) dirty exists 2023-12-15T22:59:11.310 INFO:tasks.rados.rados.0.smithi071.stdout:595: left oid 30 (ObjNum 289 snap 78 seq_num 289) 2023-12-15T22:59:11.310 INFO:tasks.rados.rados.0.smithi071.stdout:597: finishing write tid 3 to smithi07183189-6 2023-12-15T22:59:11.310 INFO:tasks.rados.rados.0.smithi071.stdout:597: finishing write tid 4 to smithi07183189-6 2023-12-15T22:59:11.310 INFO:tasks.rados.rados.0.smithi071.stdout:597: finishing write tid 5 to smithi07183189-6 2023-12-15T22:59:11.310 INFO:tasks.rados.rados.0.smithi071.stdout:597: oid 6 updating version 0 to 90 2023-12-15T22:59:11.310 INFO:tasks.rados.rados.0.smithi071.stdout:597: oid 6 version 90 is already newer than 87 2023-12-15T22:59:11.310 INFO:tasks.rados.rados.0.smithi071.stdout:597: oid 6 version 90 is already newer than 88 2023-12-15T22:59:11.310 INFO:tasks.rados.rados.0.smithi071.stdout:597: oid 6 version 90 is already newer than 89 2023-12-15T22:59:11.310 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 6 v 90 (ObjNum 291 snap 78 seq_num 291) dirty exists 2023-12-15T22:59:11.310 INFO:tasks.rados.rados.0.smithi071.stdout:597: left oid 6 (ObjNum 291 snap 78 seq_num 291) 2023-12-15T22:59:11.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:11.396+0000 7fa54ff9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:11.400 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:11.400+0000 7fa54ff9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:12.258 INFO:tasks.rados.rados.0.smithi071.stdout:590: done (5 left) 2023-12-15T22:59:12.258 INFO:tasks.rados.rados.0.smithi071.stdout:593: done (4 left) 2023-12-15T22:59:12.258 INFO:tasks.rados.rados.0.smithi071.stdout:595: done (3 left) 2023-12-15T22:59:12.259 INFO:tasks.rados.rados.0.smithi071.stdout:596: done (2 left) 2023-12-15T22:59:12.259 INFO:tasks.rados.rados.0.smithi071.stdout:597: done (1 left) 2023-12-15T22:59:12.259 INFO:tasks.rados.rados.0.smithi071.stdout:601: done (0 left) 2023-12-15T22:59:12.259 INFO:tasks.rados.rados.0.smithi071.stdout:602: snap_create 2023-12-15T22:59:12.953 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T22:59:12.953 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T22:59:12.954 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T22:59:13.070 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:13.067+0000 7f3b6e55e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:13.073 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:13.074+0000 7f3b6e55e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:13.250 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:59:13.250 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-12-15T22:59:13.258 INFO:tasks.rados.rados.0.smithi071.stdout:602: done (0 left) 2023-12-15T22:59:13.258 INFO:tasks.rados.rados.0.smithi071.stdout:603: write oid 29 current snap is 80 2023-12-15T22:59:13.258 INFO:tasks.rados.rados.0.smithi071.stdout:603: seq_num 293 ranges {524451=469371,1579224=483096,2529878=394573} 2023-12-15T22:59:13.262 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":241,"stamp":"2023-12-15T22:59:12.543484+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":110794626,"num_objects":206,"num_object_clones":141,"num_object_copies":618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":24,"num_read":1285,"num_read_kb":855244,"num_write":1122,"num_write_kb":288149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2037,"ondisk_log_size":2037,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":481928,"kb_used_data":357992,"kb_used_omap":0,"kb_used_meta":123904,"kb_avail":377005432,"statfs":{"total":386547056640,"available":386053562368,"internally_reserved":0,"allocated":366583808,"data_stored":408919869,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":126877696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2201585,"num_objects":10,"num_object_clones":10,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":34,"num_read_kb":18783,"num_write":37,"num_write_kb":10997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000780"},"pg_stats":[{"pgid":"3.f","version":"164'28","reported_seq":357,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-12-15T22:59:10.519790+0000","last_change":"2023-12-15T22:59:09.328342+0000","last_active":"2023-12-15T22:59:10.519790+0000","last_peered":"2023-12-15T22:59:10.519790+0000","last_clean":"2023-12-15T22:59:10.519790+0000","last_became_active":"2023-12-15T22:56:39.503797+0000","last_became_peered":"2023-12-15T22:56:39.503797+0000","last_unstale":"2023-12-15T22:59:10.519790+0000","last_undegraded":"2023-12-15T22:59:10.519790+0000","last_fullsized":"2023-12-15T22:59:10.519790+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4857223,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":52,"num_read_kb":39854,"num_write":21,"num_write_kb":6233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":296,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-12-15T22:59:09.340739+0000","last_change":"2023-12-15T22:59:09.343125+0000","last_active":"2023-12-15T22:59:09.340739+0000","last_peered":"2023-12-15T22:59:09.340739+0000","last_clean":"2023-12-15T22:59:09.340739+0000","last_became_active":"2023-12-15T22:56:39.552856+0000","last_became_peered":"2023-12-15T22:56:39.552856+0000","last_unstale":"2023-12-15T22:59:09.340739+0000","last_undegraded":"2023-12-15T22:59:09.340739+0000","last_fullsized":"2023-12-15T22:59:09.340739+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"158'212","reported_seq":489,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-15T22:59:06.590655+0000","last_change":"2023-12-15T22:59:06.312726+0000","last_active":"2023-12-15T22:59:06.590655+0000","last_peered":"2023-12-15T22:59:06.590655+0000","last_clean":"2023-12-15T22:59:06.590655+0000","last_became_active":"2023-12-15T22:56:39.495490+0000","last_became_peered":"2023-12-15T22:56:39.495490+0000","last_unstale":"2023-12-15T22:59:06.590655+0000","last_undegraded":"2023-12-15T22:59:06.590655+0000","last_fullsized":"2023-12-15T22:59:06.590655+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15563348,"num_objects":24,"num_object_clones":15,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":172,"num_read_kb":141222,"num_write":127,"num_write_kb":36642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"163'169","reported_seq":448,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-12-15T22:59:10.399165+0000","last_change":"2023-12-15T22:59:09.389558+0000","last_active":"2023-12-15T22:59:10.399165+0000","last_peered":"2023-12-15T22:59:10.399165+0000","last_clean":"2023-12-15T22:59:10.399165+0000","last_became_active":"2023-12-15T22:56:39.579234+0000","last_became_peered":"2023-12-15T22:56:39.579234+0000","last_unstale":"2023-12-15T22:59:10.399165+0000","last_undegraded":"2023-12-15T22:59:10.399165+0000","last_fullsized":"2023-12-15T22:59:10.399165+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7899389,"num_objects":17,"num_object_clones":14,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":115,"num_read_kb":83573,"num_write":94,"num_write_kb":26316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"164'97","reported_seq":374,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-12-15T22:59:10.331637+0000","last_change":"2023-12-15T22:59:10.358491+0000","last_active":"2023-12-15T22:59:10.331637+0000","last_peered":"2023-12-15T22:59:10.331637+0000","last_clean":"2023-12-15T22:59:10.331637+0000","last_became_active":"2023-12-15T22:56:39.572176+0000","last_became_peered":"2023-12-15T22:56:39.572176+0000","last_unstale":"2023-12-15T22:59:10.331637+0000","last_undegraded":"2023-12-15T22:59:10.331637+0000","last_fullsized":"2023-12-15T22:59:10.331637+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2756381,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":47,"num_read_kb":23792,"num_write":45,"num_write_kb":13074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"164'100","reported_seq":375,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-12-15T22:59:10.492968+0000","last_change":"2023-12-15T22:59:09.318652+0000","last_active":"2023-12-15T22:59:10.492968+0000","last_peered":"2023-12-15T22:59:10.492968+0000","last_clean":"2023-12-15T22:59:10.492968+0000","last_became_active":"2023-12-15T22:56:39.516361+0000","last_became_peered":"2023-12-15T22:56:39.516361+0000","last_unstale":"2023-12-15T22:59:10.492968+0000","last_undegraded":"2023-12-15T22:59:10.492968+0000","last_fullsized":"2023-12-15T22:59:10.492968+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5092587,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":38,"num_read_kb":25583,"num_write":51,"num_write_kb":14458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"164'381","reported_seq":558,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-12-15T22:59:10.551706+0000","last_change":"2023-12-15T22:59:09.387501+0000","last_active":"2023-12-15T22:59:10.551706+0000","last_peered":"2023-12-15T22:59:10.551706+0000","last_clean":"2023-12-15T22:59:10.551706+0000","last_became_active":"2023-12-15T22:56:39.519748+0000","last_became_peered":"2023-12-15T22:56:39.519748+0000","last_unstale":"2023-12-15T22:59:10.551706+0000","last_undegraded":"2023-12-15T22:59:10.551706+0000","last_fullsized":"2023-12-15T22:59:10.551706+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":381,"log_dups_size":0,"ondisk_log_size":381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18286148,"num_objects":41,"num_object_clones":31,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":185,"num_read_kb":113284,"num_write":176,"num_write_kb":50966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"159'85","reported_seq":378,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-15T22:59:06.292384+0000","last_change":"2023-12-15T22:59:06.303017+0000","last_active":"2023-12-15T22:59:06.292384+0000","last_peered":"2023-12-15T22:59:06.292384+0000","last_clean":"2023-12-15T22:59:06.292384+0000","last_became_active":"2023-12-15T22:56:39.543179+0000","last_became_peered":"2023-12-15T22:56:39.543179+0000","last_unstale":"2023-12-15T22:59:06.292384+0000","last_undegraded":"2023-12-15T22:59:06.292384+0000","last_fullsized":"2023-12-15T22:59:06.292384+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8369886,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":79,"num_read_kb":71705,"num_write":53,"num_write_kb":14105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":374,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-12-15T22:59:10.325801+0000","last_change":"2023-12-15T22:56:32.607017+0000","last_active":"2023-12-15T22:59:10.325801+0000","last_peered":"2023-12-15T22:59:10.325801+0000","last_clean":"2023-12-15T22:59:10.325801+0000","last_became_active":"2023-12-15T22:56:32.602384+0000","last_became_peered":"2023-12-15T22:56:32.602384+0000","last_unstale":"2023-12-15T22:59:10.325801+0000","last_undegraded":"2023-12-15T22:59:10.325801+0000","last_fullsized":"2023-12-15T22:59:10.325801+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_clean_scrub_stamp":"2023-12-15T22:56:31.525572+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":233,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-15T22:59:06.295981+0000","last_change":"2023-12-15T22:56:33.627481+0000","last_active":"2023-12-15T22:59:06.295981+0000","last_peered":"2023-12-15T22:59:06.295981+0000","last_clean":"2023-12-15T22:59:06.295981+0000","last_became_active":"2023-12-15T22:56:33.624689+0000","last_became_peered":"2023-12-15T22:56:33.624689+0000","last_unstale":"2023-12-15T22:59:06.295981+0000","last_undegraded":"2023-12-15T22:59:06.295981+0000","last_fullsized":"2023-12-15T22:59:06.295981+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"158'126","reported_seq":411,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-15T22:59:06.308402+0000","last_change":"2023-12-15T22:59:06.308402+0000","last_active":"2023-12-15T22:59:06.308402+0000","last_peered":"2023-12-15T22:59:06.308402+0000","last_clean":"2023-12-15T22:59:06.308402+0000","last_became_active":"2023-12-15T22:56:39.529359+0000","last_became_peered":"2023-12-15T22:56:39.529359+0000","last_unstale":"2023-12-15T22:59:06.308402+0000","last_undegraded":"2023-12-15T22:59:06.308402+0000","last_fullsized":"2023-12-15T22:59:06.308402+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8319498,"num_objects":13,"num_object_clones":7,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":96,"num_read_kb":68055,"num_write":75,"num_write_kb":22851,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":251,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-12-15T22:59:10.331472+0000","last_change":"2023-12-15T22:56:33.655797+0000","last_active":"2023-12-15T22:59:10.331472+0000","last_peered":"2023-12-15T22:59:10.331472+0000","last_clean":"2023-12-15T22:59:10.331472+0000","last_became_active":"2023-12-15T22:56:33.652885+0000","last_became_peered":"2023-12-15T22:56:33.652885+0000","last_unstale":"2023-12-15T22:59:10.331472+0000","last_undegraded":"2023-12-15T22:59:10.331472+0000","last_fullsized":"2023-12-15T22:59:10.331472+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"164'198","reported_seq":439,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-12-15T22:59:10.353991+0000","last_change":"2023-12-15T22:59:10.353991+0000","last_active":"2023-12-15T22:59:10.353991+0000","last_peered":"2023-12-15T22:59:10.353991+0000","last_clean":"2023-12-15T22:59:10.353991+0000","last_became_active":"2023-12-15T22:56:39.543509+0000","last_became_peered":"2023-12-15T22:56:39.543509+0000","last_unstale":"2023-12-15T22:59:10.353991+0000","last_undegraded":"2023-12-15T22:59:10.353991+0000","last_fullsized":"2023-12-15T22:59:10.353991+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5949711,"num_objects":21,"num_object_clones":15,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":111,"num_read_kb":75505,"num_write":85,"num_write_kb":25517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":233,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-15T22:59:06.297341+0000","last_change":"2023-12-15T22:56:33.600485+0000","last_active":"2023-12-15T22:59:06.297341+0000","last_peered":"2023-12-15T22:59:06.297341+0000","last_clean":"2023-12-15T22:59:06.297341+0000","last_became_active":"2023-12-15T22:56:33.597320+0000","last_became_peered":"2023-12-15T22:56:33.597320+0000","last_unstale":"2023-12-15T22:59:06.297341+0000","last_undegraded":"2023-12-15T22:59:06.297341+0000","last_fullsized":"2023-12-15T22:59:06.297341+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"164'76","reported_seq":358,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-12-15T22:59:10.514245+0000","last_change":"2023-12-15T22:59:10.336026+0000","last_active":"2023-12-15T22:59:10.514245+0000","last_peered":"2023-12-15T22:59:10.514245+0000","last_clean":"2023-12-15T22:59:10.514245+0000","last_became_active":"2023-12-15T22:56:39.560980+0000","last_became_peered":"2023-12-15T22:56:39.560980+0000","last_unstale":"2023-12-15T22:59:10.514245+0000","last_undegraded":"2023-12-15T22:59:10.514245+0000","last_fullsized":"2023-12-15T22:59:10.514245+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6605184,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":29099,"num_write":35,"num_write_kb":10458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":251,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-12-15T22:59:10.348068+0000","last_change":"2023-12-15T22:56:33.648541+0000","last_active":"2023-12-15T22:59:10.348068+0000","last_peered":"2023-12-15T22:59:10.348068+0000","last_clean":"2023-12-15T22:59:10.348068+0000","last_became_active":"2023-12-15T22:56:33.643681+0000","last_became_peered":"2023-12-15T22:56:33.643681+0000","last_unstale":"2023-12-15T22:59:10.348068+0000","last_undegraded":"2023-12-15T22:59:10.348068+0000","last_fullsized":"2023-12-15T22:59:10.348068+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"159'140","reported_seq":393,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-12-15T22:59:09.335610+0000","last_change":"2023-12-15T22:59:09.338254+0000","last_active":"2023-12-15T22:59:09.335610+0000","last_peered":"2023-12-15T22:59:09.335610+0000","last_clean":"2023-12-15T22:59:09.335610+0000","last_became_active":"2023-12-15T22:56:39.506510+0000","last_became_peered":"2023-12-15T22:56:39.506510+0000","last_unstale":"2023-12-15T22:59:09.335610+0000","last_undegraded":"2023-12-15T22:59:09.335610+0000","last_fullsized":"2023-12-15T22:59:09.335610+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8638864,"num_objects":15,"num_object_clones":10,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":80,"num_read_kb":55526,"num_write":66,"num_write_kb":19376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"158'72","reported_seq":329,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-15T22:59:06.312067+0000","last_change":"2023-12-15T22:59:06.312067+0000","last_active":"2023-12-15T22:59:06.312067+0000","last_peered":"2023-12-15T22:59:06.312067+0000","last_clean":"2023-12-15T22:59:06.312067+0000","last_became_active":"2023-12-15T22:56:39.511159+0000","last_became_peered":"2023-12-15T22:56:39.511159+0000","last_unstale":"2023-12-15T22:59:06.312067+0000","last_undegraded":"2023-12-15T22:59:06.312067+0000","last_fullsized":"2023-12-15T22:59:06.312067+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2403434,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":12,"num_read_kb":7,"num_write":33,"num_write_kb":9807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":231,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-12-15T22:59:09.335248+0000","last_change":"2023-12-15T22:56:33.633890+0000","last_active":"2023-12-15T22:59:09.335248+0000","last_peered":"2023-12-15T22:59:09.335248+0000","last_clean":"2023-12-15T22:59:09.335248+0000","last_became_active":"2023-12-15T22:56:33.631134+0000","last_became_peered":"2023-12-15T22:56:33.631134+0000","last_unstale":"2023-12-15T22:59:09.335248+0000","last_undegraded":"2023-12-15T22:59:09.335248+0000","last_fullsized":"2023-12-15T22:59:09.335248+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"157'84","reported_seq":351,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-12-15T22:59:09.337691+0000","last_change":"2023-12-15T22:59:09.341138+0000","last_active":"2023-12-15T22:59:09.337691+0000","last_peered":"2023-12-15T22:59:09.337691+0000","last_clean":"2023-12-15T22:59:09.337691+0000","last_became_active":"2023-12-15T22:56:39.532730+0000","last_became_peered":"2023-12-15T22:56:39.532730+0000","last_unstale":"2023-12-15T22:59:09.337691+0000","last_undegraded":"2023-12-15T22:59:09.337691+0000","last_fullsized":"2023-12-15T22:59:09.337691+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5092587,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":46,"num_read_kb":38481,"num_write":39,"num_write_kb":11797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":245,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-12-15T22:59:10.338209+0000","last_change":"2023-12-15T22:56:33.655736+0000","last_active":"2023-12-15T22:59:10.338209+0000","last_peered":"2023-12-15T22:59:10.338209+0000","last_clean":"2023-12-15T22:59:10.338209+0000","last_became_active":"2023-12-15T22:56:33.652917+0000","last_became_peered":"2023-12-15T22:56:33.652917+0000","last_unstale":"2023-12-15T22:59:10.338209+0000","last_undegraded":"2023-12-15T22:59:10.338209+0000","last_fullsized":"2023-12-15T22:59:10.338209+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"157'126","reported_seq":412,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-12-15T22:59:10.384766+0000","last_change":"2023-12-15T22:59:10.335559+0000","last_active":"2023-12-15T22:59:10.384766+0000","last_peered":"2023-12-15T22:59:10.384766+0000","last_clean":"2023-12-15T22:59:10.384766+0000","last_became_active":"2023-12-15T22:56:39.557252+0000","last_became_peered":"2023-12-15T22:56:39.557252+0000","last_unstale":"2023-12-15T22:59:10.384766+0000","last_undegraded":"2023-12-15T22:59:10.384766+0000","last_fullsized":"2023-12-15T22:59:10.384766+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8269110,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":80,"num_read_kb":60077,"num_write":68,"num_write_kb":19580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":231,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-12-15T22:59:09.330824+0000","last_change":"2023-12-15T22:56:33.632348+0000","last_active":"2023-12-15T22:59:09.330824+0000","last_peered":"2023-12-15T22:59:09.330824+0000","last_clean":"2023-12-15T22:59:09.330824+0000","last_became_active":"2023-12-15T22:56:33.629022+0000","last_became_peered":"2023-12-15T22:56:33.629022+0000","last_unstale":"2023-12-15T22:59:09.330824+0000","last_undegraded":"2023-12-15T22:59:09.330824+0000","last_fullsized":"2023-12-15T22:59:09.330824+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"157'46","reported_seq":325,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-12-15T22:59:09.339557+0000","last_change":"2023-12-15T22:59:09.342150+0000","last_active":"2023-12-15T22:59:09.339557+0000","last_peered":"2023-12-15T22:59:09.339557+0000","last_clean":"2023-12-15T22:59:09.339557+0000","last_became_active":"2023-12-15T22:56:39.535145+0000","last_became_peered":"2023-12-15T22:56:39.535145+0000","last_unstale":"2023-12-15T22:59:09.339557+0000","last_undegraded":"2023-12-15T22:59:09.339557+0000","last_fullsized":"2023-12-15T22:59:09.339557+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2100908,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":29,"num_read_kb":29390,"num_write":17,"num_write_kb":5075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":233,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-15T22:59:06.288963+0000","last_change":"2023-12-15T22:56:33.627005+0000","last_active":"2023-12-15T22:59:06.288963+0000","last_peered":"2023-12-15T22:59:06.288963+0000","last_clean":"2023-12-15T22:59:06.288963+0000","last_became_active":"2023-12-15T22:56:33.622181+0000","last_became_peered":"2023-12-15T22:56:33.622181+0000","last_unstale":"2023-12-15T22:59:06.288963+0000","last_undegraded":"2023-12-15T22:59:06.288963+0000","last_fullsized":"2023-12-15T22:59:06.288963+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":110204258,"num_objects":204,"num_object_clones":141,"num_object_copies":612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":24,"num_read":1182,"num_read_kb":855153,"num_write":985,"num_write_kb":286255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1940,"ondisk_log_size":1940,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705824,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":126300,"kb_used_data":95316,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94245540,"statfs":{"total":96636764160,"available":96507432960,"internally_reserved":0,"allocated":97603584,"data_stored":109652127,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705823,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":107260,"kb_used_data":76276,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94264580,"statfs":{"total":96636764160,"available":96526929920,"internally_reserved":0,"allocated":78106624,"data_stored":87802091,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705823,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":119876,"kb_used_data":88892,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94251964,"statfs":{"total":96636764160,"available":96514011136,"internally_reserved":0,"allocated":91025408,"data_stored":100684136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705824,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":128492,"kb_used_data":97508,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94243348,"statfs":{"total":96636764160,"available":96505188352,"internally_reserved":0,"allocated":99848192,"data_stored":110781515,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-15T22:59:13.265 INFO:tasks.rados.rados.0.smithi071.stdout:603: writing smithi07183189-29 from 524451 to 993822 tid 1 2023-12-15T22:59:13.272 INFO:tasks.rados.rados.0.smithi071.stdout:603: writing smithi07183189-29 from 1579224 to 2062320 tid 2 2023-12-15T22:59:13.278 INFO:tasks.rados.rados.0.smithi071.stdout:603: writing smithi07183189-29 from 2529878 to 2924451 tid 3 2023-12-15T22:59:13.278 INFO:tasks.rados.rados.0.smithi071.stdout:604: write (excl) oid 19 current snap is 80 2023-12-15T22:59:13.278 INFO:tasks.rados.rados.0.smithi071.stdout:604: seq_num 294 ranges {541258=544620,1637706=626754,2840948=100310} 2023-12-15T22:59:13.285 INFO:tasks.rados.rados.0.smithi071.stdout:604: writing smithi07183189-19 from 541258 to 1085878 tid 1 2023-12-15T22:59:13.294 INFO:tasks.rados.rados.0.smithi071.stdout:604: writing smithi07183189-19 from 1637706 to 2264460 tid 2 2023-12-15T22:59:13.298 INFO:tasks.rados.rados.0.smithi071.stdout:604: writing smithi07183189-19 from 2840948 to 2941258 tid 3 2023-12-15T22:59:13.298 INFO:tasks.rados.rados.0.smithi071.stdout:605: write (excl) oid 31 current snap is 80 2023-12-15T22:59:13.298 INFO:tasks.rados.rados.0.smithi071.stdout:605: seq_num 295 ranges {558065=619869,1579835=486765,2468371=489694} 2023-12-15T22:59:13.306 INFO:tasks.rados.rados.0.smithi071.stdout:605: writing smithi07183189-31 from 558065 to 1177934 tid 1 2023-12-15T22:59:13.313 INFO:tasks.rados.rados.0.smithi071.stdout:605: writing smithi07183189-31 from 1579835 to 2066600 tid 2 2023-12-15T22:59:13.320 INFO:tasks.rados.rados.0.smithi071.stdout:605: writing smithi07183189-31 from 2468371 to 2958065 tid 3 2023-12-15T22:59:13.320 INFO:tasks.rados.rados.0.smithi071.stdout:606: read oid 22 snap -1 2023-12-15T22:59:13.320 INFO:tasks.rados.rados.0.smithi071.stdout:606: expect (ObjNum 276 snap 72 seq_num 276) 2023-12-15T22:59:13.320 INFO:tasks.rados.rados.0.smithi071.stdout:607: read oid 12 snap 51 2023-12-15T22:59:13.320 INFO:tasks.rados.rados.0.smithi071.stdout:607: expect deleted 2023-12-15T22:59:13.320 INFO:tasks.rados.rados.0.smithi071.stdout:608: read oid 9 snap -1 2023-12-15T22:59:13.320 INFO:tasks.rados.rados.0.smithi071.stdout:608: expect (ObjNum 271 snap 70 seq_num 271) 2023-12-15T22:59:13.320 INFO:tasks.rados.rados.0.smithi071.stdout:609: write (excl) oid 24 current snap is 80 2023-12-15T22:59:13.320 INFO:tasks.rados.rados.0.smithi071.stdout:609: seq_num 296 ranges {574872=695118,2038317=630423} 2023-12-15T22:59:13.327 INFO:tasks.rados.rados.0.smithi071.stdout:609: writing smithi07183189-24 from 574872 to 1269990 tid 1 2023-12-15T22:59:13.336 INFO:tasks.rados.rados.0.smithi071.stdout:609: writing smithi07183189-24 from 2038317 to 2668740 tid 2 2023-12-15T22:59:13.336 INFO:tasks.rados.rados.0.smithi071.stdout:610: read oid 41 snap -1 2023-12-15T22:59:13.336 INFO:tasks.rados.rados.0.smithi071.stdout:610: expect (ObjNum 241 snap 64 seq_num 241) 2023-12-15T22:59:13.336 INFO:tasks.rados.rados.0.smithi071.stdout:611: read oid 17 snap -1 2023-12-15T22:59:13.336 INFO:tasks.rados.rados.0.smithi071.stdout:611: expect (ObjNum 220 snap 49 seq_num 220) 2023-12-15T22:59:13.336 INFO:tasks.rados.rados.0.smithi071.stdout:612: delete oid 40 current snap is 80 2023-12-15T22:59:13.376 INFO:tasks.rados.rados.0.smithi071.stdout:603: finishing write tid 1 to smithi07183189-29 2023-12-15T22:59:13.385 INFO:tasks.rados.rados.0.smithi071.stdout:611: expect (ObjNum 220 snap 49 seq_num 220) 2023-12-15T22:59:13.470 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:59:13.581 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:13.580+0000 7fbc2b15e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:13.584 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:13.584+0000 7fbc2b15e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:13.728 INFO:tasks.rados.rados.0.smithi071.stdout:604: finishing write tid 1 to smithi07183189-19 2023-12-15T22:59:13.728 INFO:tasks.rados.rados.0.smithi071.stdout:606: expect (ObjNum 276 snap 72 seq_num 276) 2023-12-15T22:59:13.884 INFO:tasks.rados.rados.0.smithi071.stdout:609: finishing write tid 1 to smithi07183189-24 2023-12-15T22:59:13.884 INFO:tasks.rados.rados.0.smithi071.stdout:605: finishing write tid 1 to smithi07183189-31 2023-12-15T22:59:13.884 INFO:tasks.rados.rados.0.smithi071.stdout:603: finishing write tid 2 to smithi07183189-29 2023-12-15T22:59:13.884 INFO:tasks.rados.rados.0.smithi071.stdout:610: expect (ObjNum 241 snap 64 seq_num 241) 2023-12-15T22:59:14.002 INFO:tasks.rados.rados.0.smithi071.stdout:604: finishing write tid 2 to smithi07183189-19 2023-12-15T22:59:14.002 INFO:tasks.rados.rados.0.smithi071.stdout:609: finishing write tid 2 to smithi07183189-24 2023-12-15T22:59:14.003 INFO:tasks.rados.rados.0.smithi071.stdout:605: finishing write tid 2 to smithi07183189-31 2023-12-15T22:59:14.003 INFO:tasks.rados.rados.0.smithi071.stdout:603: finishing write tid 3 to smithi07183189-29 2023-12-15T22:59:14.003 INFO:tasks.rados.rados.0.smithi071.stdout:604: finishing write tid 3 to smithi07183189-19 2023-12-15T22:59:14.003 INFO:tasks.rados.rados.0.smithi071.stdout:609: finishing write tid 3 to smithi07183189-24 2023-12-15T22:59:14.003 INFO:tasks.rados.rados.0.smithi071.stdout:605: finishing write tid 3 to smithi07183189-31 2023-12-15T22:59:14.003 INFO:tasks.rados.rados.0.smithi071.stdout:609: finishing write tid 4 to smithi07183189-24 2023-12-15T22:59:14.003 INFO:tasks.rados.rados.0.smithi071.stdout:609: oid 24 updating version 0 to 74 2023-12-15T22:59:14.003 INFO:tasks.rados.rados.0.smithi071.stdout:609: oid 24 updating version 74 to 76 2023-12-15T22:59:14.003 INFO:tasks.rados.rados.0.smithi071.stdout:609: oid 24 version 76 is already newer than 76 2023-12-15T22:59:14.003 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 76 (ObjNum 296 snap 80 seq_num 296) dirty exists 2023-12-15T22:59:14.003 INFO:tasks.rados.rados.0.smithi071.stdout:609: left oid 24 (ObjNum 296 snap 80 seq_num 296) 2023-12-15T22:59:14.003 INFO:tasks.rados.rados.0.smithi071.stdout:605: finishing write tid 4 to smithi07183189-31 2023-12-15T22:59:14.003 INFO:tasks.rados.rados.0.smithi071.stdout:606: done (9 left) 2023-12-15T22:59:14.003 INFO:tasks.rados.rados.0.smithi071.stdout:607: done (8 left) 2023-12-15T22:59:14.003 INFO:tasks.rados.rados.0.smithi071.stdout:609: done (7 left) 2023-12-15T22:59:14.003 INFO:tasks.rados.rados.0.smithi071.stdout:610: done (6 left) 2023-12-15T22:59:14.003 INFO:tasks.rados.rados.0.smithi071.stdout:611: done (5 left) 2023-12-15T22:59:14.004 INFO:tasks.rados.rados.0.smithi071.stdout:612: done (4 left) 2023-12-15T22:59:14.004 INFO:tasks.rados.rados.0.smithi071.stdout:613: read oid 36 snap 67 2023-12-15T22:59:14.004 INFO:tasks.rados.rados.0.smithi071.stdout:613: expect (ObjNum 226 snap 54 seq_num 226) 2023-12-15T22:59:14.004 INFO:tasks.rados.rados.0.smithi071.stdout:614: write (excl) oid 27 current snap is 80 2023-12-15T22:59:14.004 INFO:tasks.rados.rados.0.smithi071.stdout:605: finishing write tid 5 to smithi07183189-31 2023-12-15T22:59:14.004 INFO:tasks.rados.rados.0.smithi071.stdout:605: oid 31 updating version 0 to 130 2023-12-15T22:59:14.004 INFO:tasks.rados.rados.0.smithi071.stdout:605: oid 31 updating version 130 to 131 2023-12-15T22:59:14.004 INFO:tasks.rados.rados.0.smithi071.stdout:605: oid 31 version 131 is already newer than 130 2023-12-15T22:59:14.004 INFO:tasks.rados.rados.0.smithi071.stdout:605: oid 31 version 131 is already newer than 128 2023-12-15T22:59:14.004 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 131 (ObjNum 295 snap 80 seq_num 295) dirty exists 2023-12-15T22:59:14.004 INFO:tasks.rados.rados.0.smithi071.stdout:605: left oid 31 (ObjNum 295 snap 80 seq_num 295) 2023-12-15T22:59:14.004 INFO:tasks.rados.rados.0.smithi071.stdout:604: finishing write tid 4 to smithi07183189-19 2023-12-15T22:59:14.004 INFO:tasks.rados.rados.0.smithi071.stdout:603: finishing write tid 4 to smithi07183189-29 2023-12-15T22:59:14.004 INFO:tasks.rados.rados.0.smithi071.stdout:603: finishing write tid 5 to smithi07183189-29 2023-12-15T22:59:14.004 INFO:tasks.rados.rados.0.smithi071.stdout:603: oid 29 updating version 0 to 391 2023-12-15T22:59:14.005 INFO:tasks.rados.rados.0.smithi071.stdout:603: oid 29 updating version 391 to 392 2023-12-15T22:59:14.005 INFO:tasks.rados.rados.0.smithi071.stdout:603: oid 29 version 392 is already newer than 390 2023-12-15T22:59:14.005 INFO:tasks.rados.rados.0.smithi071.stdout:603: oid 29 version 392 is already newer than 389 2023-12-15T22:59:14.005 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 392 (ObjNum 293 snap 80 seq_num 293) dirty exists 2023-12-15T22:59:14.005 INFO:tasks.rados.rados.0.smithi071.stdout:603: left oid 29 (ObjNum 293 snap 80 seq_num 293) 2023-12-15T22:59:14.005 INFO:tasks.rados.rados.0.smithi071.stdout:604: finishing write tid 5 to smithi07183189-19 2023-12-15T22:59:14.005 INFO:tasks.rados.rados.0.smithi071.stdout:604: oid 19 updating version 0 to 175 2023-12-15T22:59:14.005 INFO:tasks.rados.rados.0.smithi071.stdout:604: oid 19 version 175 is already newer than 174 2023-12-15T22:59:14.005 INFO:tasks.rados.rados.0.smithi071.stdout:604: oid 19 updating version 175 to 176 2023-12-15T22:59:14.005 INFO:tasks.rados.rados.0.smithi071.stdout:604: oid 19 version 176 is already newer than 173 2023-12-15T22:59:14.005 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 176 (ObjNum 294 snap 80 seq_num 294) dirty exists 2023-12-15T22:59:14.005 INFO:tasks.rados.rados.0.smithi071.stdout:604: left oid 19 (ObjNum 294 snap 80 seq_num 294) 2023-12-15T22:59:14.005 INFO:tasks.rados.rados.0.smithi071.stdout:614: seq_num 298 ranges {608486=561969,1871634=634092,2950640=57846} 2023-12-15T22:59:14.009 INFO:tasks.rados.rados.0.smithi071.stdout:614: writing smithi07183189-27 from 608486 to 1170455 tid 1 2023-12-15T22:59:14.015 INFO:tasks.rados.rados.0.smithi071.stdout:614: writing smithi07183189-27 from 1871634 to 2505726 tid 2 2023-12-15T22:59:14.018 INFO:tasks.rados.rados.0.smithi071.stdout:614: writing smithi07183189-27 from 2950640 to 3008486 tid 3 2023-12-15T22:59:14.018 INFO:tasks.rados.rados.0.smithi071.stdout:603: done (5 left) 2023-12-15T22:59:14.018 INFO:tasks.rados.rados.0.smithi071.stdout:604: done (4 left) 2023-12-15T22:59:14.018 INFO:tasks.rados.rados.0.smithi071.stdout:605: done (3 left) 2023-12-15T22:59:14.018 INFO:tasks.rados.rados.0.smithi071.stdout:615: delete oid 18 current snap is 80 2023-12-15T22:59:14.018 INFO:tasks.rados.rados.0.smithi071.stdout:608: expect (ObjNum 271 snap 70 seq_num 271) 2023-12-15T22:59:14.215 INFO:tasks.rados.rados.0.smithi071.stdout:613: expect (ObjNum 226 snap 54 seq_num 226) 2023-12-15T22:59:14.410 INFO:tasks.rados.rados.0.smithi071.stdout:614: finishing write tid 1 to smithi07183189-27 2023-12-15T22:59:14.411 INFO:tasks.rados.rados.0.smithi071.stdout:614: finishing write tid 2 to smithi07183189-27 2023-12-15T22:59:14.411 INFO:tasks.rados.rados.0.smithi071.stdout:614: finishing write tid 3 to smithi07183189-27 2023-12-15T22:59:14.411 INFO:tasks.rados.rados.0.smithi071.stdout:614: finishing write tid 4 to smithi07183189-27 2023-12-15T22:59:14.411 INFO:tasks.rados.rados.0.smithi071.stdout:614: finishing write tid 5 to smithi07183189-27 2023-12-15T22:59:14.411 INFO:tasks.rados.rados.0.smithi071.stdout:614: oid 27 updating version 0 to 147 2023-12-15T22:59:14.411 INFO:tasks.rados.rados.0.smithi071.stdout:614: oid 27 version 147 is already newer than 146 2023-12-15T22:59:14.411 INFO:tasks.rados.rados.0.smithi071.stdout:614: oid 27 version 147 is already newer than 144 2023-12-15T22:59:14.411 INFO:tasks.rados.rados.0.smithi071.stdout:614: oid 27 version 147 is already newer than 146 2023-12-15T22:59:14.411 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 147 (ObjNum 298 snap 80 seq_num 298) dirty exists 2023-12-15T22:59:14.411 INFO:tasks.rados.rados.0.smithi071.stdout:614: left oid 27 (ObjNum 298 snap 80 seq_num 298) 2023-12-15T22:59:14.411 INFO:tasks.rados.rados.0.smithi071.stdout:608: done (3 left) 2023-12-15T22:59:14.411 INFO:tasks.rados.rados.0.smithi071.stdout:613: done (2 left) 2023-12-15T22:59:14.411 INFO:tasks.rados.rados.0.smithi071.stdout:614: done (1 left) 2023-12-15T22:59:14.411 INFO:tasks.rados.rados.0.smithi071.stdout:615: done (0 left) 2023-12-15T22:59:14.412 INFO:tasks.rados.rados.0.smithi071.stdout:616: write oid 6 current snap is 80 2023-12-15T22:59:14.412 INFO:tasks.rados.rados.0.smithi071.stdout:616: seq_num 300 ranges {642100=712467,1872245=637761} 2023-12-15T22:59:14.417 INFO:tasks.rados.rados.0.smithi071.stdout:616: writing smithi07183189-6 from 642100 to 1354567 tid 1 2023-12-15T22:59:14.424 INFO:tasks.rados.rados.0.smithi071.stdout:616: writing smithi07183189-6 from 1872245 to 2510006 tid 2 2023-12-15T22:59:14.424 INFO:tasks.rados.rados.0.smithi071.stdout:617: write oid 50 current snap is 80 2023-12-15T22:59:14.424 INFO:tasks.rados.rados.0.smithi071.stdout:617: seq_num 301 ranges {658907=787716,1930727=497772,3032909=25998} 2023-12-15T22:59:14.432 INFO:tasks.rados.rados.0.smithi071.stdout:617: writing smithi07183189-50 from 658907 to 1446623 tid 1 2023-12-15T22:59:14.439 INFO:tasks.rados.rados.0.smithi071.stdout:617: writing smithi07183189-50 from 1930727 to 2428499 tid 2 2023-12-15T22:59:14.443 INFO:tasks.rados.rados.0.smithi071.stdout:617: writing smithi07183189-50 from 3032909 to 3058907 tid 3 2023-12-15T22:59:14.443 INFO:tasks.rados.rados.0.smithi071.stdout:618: delete oid 33 current snap is 80 2023-12-15T22:59:14.461 INFO:tasks.rados.rados.0.smithi071.stdout:616: finishing write tid 1 to smithi07183189-6 2023-12-15T22:59:14.472 INFO:tasks.rados.rados.0.smithi071.stdout:617: finishing write tid 1 to smithi07183189-50 2023-12-15T22:59:14.483 INFO:tasks.rados.rados.0.smithi071.stdout:616: finishing write tid 2 to smithi07183189-6 2023-12-15T22:59:14.485 INFO:tasks.rados.rados.0.smithi071.stdout:616: finishing write tid 3 to smithi07183189-6 2023-12-15T22:59:14.486 INFO:tasks.rados.rados.0.smithi071.stdout:616: finishing write tid 4 to smithi07183189-6 2023-12-15T22:59:14.487 INFO:tasks.rados.rados.0.smithi071.stdout:616: oid 6 updating version 0 to 96 2023-12-15T22:59:14.487 INFO:tasks.rados.rados.0.smithi071.stdout:616: oid 6 version 96 is already newer than 96 2023-12-15T22:59:14.487 INFO:tasks.rados.rados.0.smithi071.stdout:616: oid 6 version 96 is already newer than 94 2023-12-15T22:59:14.487 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 6 v 96 (ObjNum 300 snap 80 seq_num 300) dirty exists 2023-12-15T22:59:14.487 INFO:tasks.rados.rados.0.smithi071.stdout:616: left oid 6 (ObjNum 300 snap 80 seq_num 300) 2023-12-15T22:59:14.489 INFO:tasks.rados.rados.0.smithi071.stdout:617: finishing write tid 2 to smithi07183189-50 2023-12-15T22:59:14.499 INFO:tasks.rados.rados.0.smithi071.stdout:617: finishing write tid 3 to smithi07183189-50 2023-12-15T22:59:14.507 INFO:tasks.rados.rados.0.smithi071.stdout:617: finishing write tid 4 to smithi07183189-50 2023-12-15T22:59:14.509 INFO:tasks.rados.rados.0.smithi071.stdout:617: finishing write tid 5 to smithi07183189-50 2023-12-15T22:59:14.509 INFO:tasks.rados.rados.0.smithi071.stdout:617: oid 50 updating version 0 to 396 2023-12-15T22:59:14.509 INFO:tasks.rados.rados.0.smithi071.stdout:617: oid 50 updating version 396 to 397 2023-12-15T22:59:14.510 INFO:tasks.rados.rados.0.smithi071.stdout:617: oid 50 updating version 397 to 399 2023-12-15T22:59:14.510 INFO:tasks.rados.rados.0.smithi071.stdout:617: oid 50 version 399 is already newer than 398 2023-12-15T22:59:14.510 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 399 (ObjNum 301 snap 80 seq_num 301) dirty exists 2023-12-15T22:59:14.510 INFO:tasks.rados.rados.0.smithi071.stdout:617: left oid 50 (ObjNum 301 snap 80 seq_num 301) 2023-12-15T22:59:14.524 INFO:tasks.rados.rados.0.smithi071.stdout:616: done (2 left) 2023-12-15T22:59:14.524 INFO:tasks.rados.rados.0.smithi071.stdout:617: done (1 left) 2023-12-15T22:59:14.525 INFO:tasks.rados.rados.0.smithi071.stdout:618: done (0 left) 2023-12-15T22:59:14.525 INFO:tasks.rados.rados.0.smithi071.stdout:619: write oid 45 current snap is 80 2023-12-15T22:59:14.525 INFO:tasks.rados.rados.0.smithi071.stdout:619: seq_num 303 ranges {692521=538214,1931338=785088} 2023-12-15T22:59:14.531 INFO:tasks.rados.rados.0.smithi071.stdout:619: writing smithi07183189-45 from 692521 to 1230735 tid 1 2023-12-15T22:59:14.541 INFO:tasks.rados.rados.0.smithi071.stdout:619: writing smithi07183189-45 from 1931338 to 2716426 tid 2 2023-12-15T22:59:14.541 INFO:tasks.rados.rados.0.smithi071.stdout:620: read oid 2 snap -1 2023-12-15T22:59:14.541 INFO:tasks.rados.rados.0.smithi071.stdout:620: expect (ObjNum 286 snap 78 seq_num 286) 2023-12-15T22:59:14.541 INFO:tasks.rados.rados.0.smithi071.stdout:621: write (excl) oid 1 current snap is 80 2023-12-15T22:59:14.541 INFO:tasks.rados.rados.0.smithi071.stdout:621: seq_num 304 ranges {709328=613463,1989820=645099} 2023-12-15T22:59:14.549 INFO:tasks.rados.rados.0.smithi071.stdout:621: writing smithi07183189-1 from 709328 to 1322791 tid 1 2023-12-15T22:59:14.558 INFO:tasks.rados.rados.0.smithi071.stdout:621: writing smithi07183189-1 from 1989820 to 2634919 tid 2 2023-12-15T22:59:14.558 INFO:tasks.rados.rados.0.smithi071.stdout:622: read oid 11 snap 78 2023-12-15T22:59:14.558 INFO:tasks.rados.rados.0.smithi071.stdout:622: expect (ObjNum 270 snap 70 seq_num 270) 2023-12-15T22:59:14.558 INFO:tasks.rados.rados.0.smithi071.stdout:623: delete oid 7 current snap is 80 2023-12-15T22:59:14.586 INFO:tasks.rados.rados.0.smithi071.stdout:622: expect (ObjNum 270 snap 70 seq_num 270) 2023-12-15T22:59:14.736 INFO:tasks.rados.rados.0.smithi071.stdout:620: expect (ObjNum 286 snap 78 seq_num 286) 2023-12-15T22:59:14.879 INFO:tasks.rados.rados.0.smithi071.stdout:619: finishing write tid 1 to smithi07183189-45 2023-12-15T22:59:14.880 INFO:tasks.rados.rados.0.smithi071.stdout:621: finishing write tid 1 to smithi07183189-1 2023-12-15T22:59:14.880 INFO:tasks.rados.rados.0.smithi071.stdout:619: finishing write tid 2 to smithi07183189-45 2023-12-15T22:59:14.880 INFO:tasks.rados.rados.0.smithi071.stdout:621: finishing write tid 2 to smithi07183189-1 2023-12-15T22:59:14.880 INFO:tasks.rados.rados.0.smithi071.stdout:619: finishing write tid 3 to smithi07183189-45 2023-12-15T22:59:14.880 INFO:tasks.rados.rados.0.smithi071.stdout:619: finishing write tid 4 to smithi07183189-45 2023-12-15T22:59:14.880 INFO:tasks.rados.rados.0.smithi071.stdout:619: oid 45 updating version 0 to 404 2023-12-15T22:59:14.880 INFO:tasks.rados.rados.0.smithi071.stdout:619: oid 45 version 404 is already newer than 403 2023-12-15T22:59:14.880 INFO:tasks.rados.rados.0.smithi071.stdout:619: oid 45 updating version 404 to 405 2023-12-15T22:59:14.880 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 405 (ObjNum 303 snap 80 seq_num 303) dirty exists 2023-12-15T22:59:14.880 INFO:tasks.rados.rados.0.smithi071.stdout:619: left oid 45 (ObjNum 303 snap 80 seq_num 303) 2023-12-15T22:59:14.880 INFO:tasks.rados.rados.0.smithi071.stdout:621: finishing write tid 3 to smithi07183189-1 2023-12-15T22:59:14.880 INFO:tasks.rados.rados.0.smithi071.stdout:619: done (4 left) 2023-12-15T22:59:14.880 INFO:tasks.rados.rados.0.smithi071.stdout:620: done (3 left) 2023-12-15T22:59:14.881 INFO:tasks.rados.rados.0.smithi071.stdout:622: done (2 left) 2023-12-15T22:59:14.881 INFO:tasks.rados.rados.0.smithi071.stdout:623: done (1 left) 2023-12-15T22:59:14.881 INFO:tasks.rados.rados.0.smithi071.stdout:624: write oid 11 current snap is 80 2023-12-15T22:59:14.881 INFO:tasks.rados.rados.0.smithi071.stdout:621: finishing write tid 4 to smithi07183189-1 2023-12-15T22:59:14.881 INFO:tasks.rados.rados.0.smithi071.stdout:621: oid 1 updating version 0 to 33 2023-12-15T22:59:14.881 INFO:tasks.rados.rados.0.smithi071.stdout:621: oid 1 version 33 is already newer than 32 2023-12-15T22:59:14.881 INFO:tasks.rados.rados.0.smithi071.stdout:621: oid 1 updating version 33 to 34 2023-12-15T22:59:14.881 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 1 v 34 (ObjNum 304 snap 80 seq_num 304) dirty exists 2023-12-15T22:59:14.881 INFO:tasks.rados.rados.0.smithi071.stdout:621: left oid 1 (ObjNum 304 snap 80 seq_num 304) 2023-12-15T22:59:14.881 INFO:tasks.rados.rados.0.smithi071.stdout:624: seq_num 306 ranges {742942=480314,1823137=648768} 2023-12-15T22:59:14.886 INFO:tasks.rados.rados.0.smithi071.stdout:624: writing smithi07183189-11 from 742942 to 1223256 tid 1 2023-12-15T22:59:14.894 INFO:tasks.rados.rados.0.smithi071.stdout:624: writing smithi07183189-11 from 1823137 to 2471905 tid 2 2023-12-15T22:59:14.895 INFO:tasks.rados.rados.0.smithi071.stdout:621: done (1 left) 2023-12-15T22:59:14.895 INFO:tasks.rados.rados.0.smithi071.stdout:625: read oid 40 snap -1 2023-12-15T22:59:14.895 INFO:tasks.rados.rados.0.smithi071.stdout:625: expect deleted 2023-12-15T22:59:14.895 INFO:tasks.rados.rados.0.smithi071.stdout:626: snap_create 2023-12-15T22:59:14.916 INFO:tasks.rados.rados.0.smithi071.stdout:624: finishing write tid 1 to smithi07183189-11 2023-12-15T22:59:14.930 INFO:tasks.rados.rados.0.smithi071.stdout:624: finishing write tid 2 to smithi07183189-11 2023-12-15T22:59:14.937 INFO:tasks.rados.rados.0.smithi071.stdout:624: finishing write tid 3 to smithi07183189-11 2023-12-15T22:59:14.938 INFO:tasks.rados.rados.0.smithi071.stdout:624: finishing write tid 4 to smithi07183189-11 2023-12-15T22:59:14.938 INFO:tasks.rados.rados.0.smithi071.stdout:624: oid 11 updating version 0 to 129 2023-12-15T22:59:14.938 INFO:tasks.rados.rados.0.smithi071.stdout:624: oid 11 updating version 129 to 130 2023-12-15T22:59:14.938 INFO:tasks.rados.rados.0.smithi071.stdout:624: oid 11 version 130 is already newer than 128 2023-12-15T22:59:14.938 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 11 v 130 (ObjNum 306 snap 80 seq_num 306) dirty exists 2023-12-15T22:59:14.938 INFO:tasks.rados.rados.0.smithi071.stdout:624: left oid 11 (ObjNum 306 snap 80 seq_num 306) 2023-12-15T22:59:15.267 INFO:tasks.rados.rados.0.smithi071.stdout:624: done (2 left) 2023-12-15T22:59:15.267 INFO:tasks.rados.rados.0.smithi071.stdout:625: done (1 left) 2023-12-15T22:59:15.268 INFO:tasks.rados.rados.0.smithi071.stdout:626: done (0 left) 2023-12-15T22:59:15.268 INFO:tasks.rados.rados.0.smithi071.stdout:627: delete oid 4 current snap is 81 2023-12-15T22:59:15.292 INFO:tasks.rados.rados.0.smithi071.stdout:627: done (0 left) 2023-12-15T22:59:15.293 INFO:tasks.rados.rados.0.smithi071.stdout:628: delete oid 1 current snap is 81 2023-12-15T22:59:15.326 INFO:tasks.rados.rados.0.smithi071.stdout:628: done (0 left) 2023-12-15T22:59:15.326 INFO:tasks.rados.rados.0.smithi071.stdout:629: read oid 41 snap 51 2023-12-15T22:59:15.327 INFO:tasks.rados.rados.0.smithi071.stdout:629: expect (ObjNum 113 snap 10 seq_num 113) 2023-12-15T22:59:15.327 INFO:tasks.rados.rados.0.smithi071.stdout:630: read oid 28 snap -1 2023-12-15T22:59:15.327 INFO:tasks.rados.rados.0.smithi071.stdout:630: expect (ObjNum 283 snap 76 seq_num 283) 2023-12-15T22:59:15.327 INFO:tasks.rados.rados.0.smithi071.stdout:631: snap_remove snap 67 2023-12-15T22:59:15.668 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:59:15.773 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:15.770+0000 7fe01622d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:15.776 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:15.775+0000 7fe01622d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:16.266 INFO:tasks.rados.rados.0.smithi071.stdout:631: done (2 left) 2023-12-15T22:59:16.267 INFO:tasks.rados.rados.0.smithi071.stdout:632: delete oid 4 current snap is 81 2023-12-15T22:59:16.267 INFO:tasks.rados.rados.0.smithi071.stdout:630: expect (ObjNum 283 snap 76 seq_num 283) 2023-12-15T22:59:16.428 INFO:tasks.rados.rados.0.smithi071.stdout:629: expect (ObjNum 113 snap 10 seq_num 113) 2023-12-15T22:59:16.649 INFO:tasks.rados.rados.0.smithi071.stdout:629: done (2 left) 2023-12-15T22:59:16.650 INFO:tasks.rados.rados.0.smithi071.stdout:630: done (1 left) 2023-12-15T22:59:16.650 INFO:tasks.rados.rados.0.smithi071.stdout:632: done (0 left) 2023-12-15T22:59:16.650 INFO:tasks.rados.rados.0.smithi071.stdout:633: write oid 44 current snap is 81 2023-12-15T22:59:16.650 INFO:tasks.rados.rados.0.smithi071.stdout:633: seq_num 310 ranges {410170=781310,1940712=656106,3163363=46807} 2023-12-15T22:59:16.657 INFO:tasks.rados.rados.0.smithi071.stdout:633: writing smithi07183189-44 from 410170 to 1191480 tid 1 2023-12-15T22:59:16.665 INFO:tasks.rados.rados.0.smithi071.stdout:633: writing smithi07183189-44 from 1940712 to 2596818 tid 2 2023-12-15T22:59:16.669 INFO:tasks.rados.rados.0.smithi071.stdout:633: writing smithi07183189-44 from 3163363 to 3210170 tid 3 2023-12-15T22:59:16.669 INFO:tasks.rados.rados.0.smithi071.stdout:634: delete oid 19 current snap is 81 2023-12-15T22:59:16.701 INFO:tasks.rados.rados.0.smithi071.stdout:633: finishing write tid 1 to smithi07183189-44 2023-12-15T22:59:16.721 INFO:tasks.rados.rados.0.smithi071.stdout:634: done (1 left) 2023-12-15T22:59:16.721 INFO:tasks.rados.rados.0.smithi071.stdout:635: read oid 42 snap -1 2023-12-15T22:59:16.721 INFO:tasks.rados.rados.0.smithi071.stdout:635: expect deleted 2023-12-15T22:59:16.722 INFO:tasks.rados.rados.0.smithi071.stdout:636: read oid 13 snap -1 2023-12-15T22:59:16.722 INFO:tasks.rados.rados.0.smithi071.stdout:636: expect deleted 2023-12-15T22:59:16.722 INFO:tasks.rados.rados.0.smithi071.stdout:637: read oid 15 snap -1 2023-12-15T22:59:16.722 INFO:tasks.rados.rados.0.smithi071.stdout:637: expect (ObjNum 197 snap 40 seq_num 197) 2023-12-15T22:59:16.722 INFO:tasks.rados.rados.0.smithi071.stdout:638: read oid 22 snap -1 2023-12-15T22:59:16.722 INFO:tasks.rados.rados.0.smithi071.stdout:638: expect (ObjNum 276 snap 72 seq_num 276) 2023-12-15T22:59:16.722 INFO:tasks.rados.rados.0.smithi071.stdout:639: write (excl) oid 32 current snap is 81 2023-12-15T22:59:16.722 INFO:tasks.rados.rados.0.smithi071.stdout:639: seq_num 312 ranges {443784=648161,1657676=659775,2818209=425575} 2023-12-15T22:59:16.728 INFO:tasks.rados.rados.0.smithi071.stdout:639: writing smithi07183189-32 from 443784 to 1091945 tid 1 2023-12-15T22:59:16.736 INFO:tasks.rados.rados.0.smithi071.stdout:639: writing smithi07183189-32 from 1657676 to 2317451 tid 2 2023-12-15T22:59:16.743 INFO:tasks.rados.rados.0.smithi071.stdout:639: writing smithi07183189-32 from 2818209 to 3243784 tid 3 2023-12-15T22:59:16.743 INFO:tasks.rados.rados.0.smithi071.stdout:640: delete oid 31 current snap is 81 2023-12-15T22:59:16.743 INFO:tasks.rados.rados.0.smithi071.stdout:633: finishing write tid 2 to smithi07183189-44 2023-12-15T22:59:16.743 INFO:tasks.rados.rados.0.smithi071.stdout:633: finishing write tid 3 to smithi07183189-44 2023-12-15T22:59:16.744 INFO:tasks.rados.rados.0.smithi071.stdout:637: expect (ObjNum 197 snap 40 seq_num 197) 2023-12-15T22:59:16.897 INFO:tasks.rados.rados.0.smithi071.stdout:638: expect (ObjNum 276 snap 72 seq_num 276) 2023-12-15T22:59:17.054 INFO:tasks.rados.rados.0.smithi071.stdout:633: finishing write tid 4 to smithi07183189-44 2023-12-15T22:59:17.054 INFO:tasks.rados.rados.0.smithi071.stdout:633: finishing write tid 5 to smithi07183189-44 2023-12-15T22:59:17.054 INFO:tasks.rados.rados.0.smithi071.stdout:633: oid 44 updating version 0 to 88 2023-12-15T22:59:17.054 INFO:tasks.rados.rados.0.smithi071.stdout:633: oid 44 updating version 88 to 90 2023-12-15T22:59:17.054 INFO:tasks.rados.rados.0.smithi071.stdout:633: oid 44 updating version 90 to 91 2023-12-15T22:59:17.054 INFO:tasks.rados.rados.0.smithi071.stdout:633: oid 44 version 91 is already newer than 90 2023-12-15T22:59:17.054 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 91 (ObjNum 310 snap 81 seq_num 310) dirty exists 2023-12-15T22:59:17.055 INFO:tasks.rados.rados.0.smithi071.stdout:633: left oid 44 (ObjNum 310 snap 81 seq_num 310) 2023-12-15T22:59:17.055 INFO:tasks.rados.rados.0.smithi071.stdout:639: finishing write tid 1 to smithi07183189-32 2023-12-15T22:59:17.055 INFO:tasks.rados.rados.0.smithi071.stdout:633: done (6 left) 2023-12-15T22:59:17.055 INFO:tasks.rados.rados.0.smithi071.stdout:635: done (5 left) 2023-12-15T22:59:17.055 INFO:tasks.rados.rados.0.smithi071.stdout:636: done (4 left) 2023-12-15T22:59:17.055 INFO:tasks.rados.rados.0.smithi071.stdout:637: done (3 left) 2023-12-15T22:59:17.055 INFO:tasks.rados.rados.0.smithi071.stdout:638: done (2 left) 2023-12-15T22:59:17.055 INFO:tasks.rados.rados.0.smithi071.stdout:640: done (1 left) 2023-12-15T22:59:17.055 INFO:tasks.rados.rados.0.smithi071.stdout:641: read oid 39 snap 63 2023-12-15T22:59:17.055 INFO:tasks.rados.rados.0.smithi071.stdout:641: expect deleted 2023-12-15T22:59:17.055 INFO:tasks.rados.rados.0.smithi071.stdout:642: delete oid 45 current snap is 81 2023-12-15T22:59:17.055 INFO:tasks.rados.rados.0.smithi071.stdout:639: finishing write tid 2 to smithi07183189-32 2023-12-15T22:59:17.055 INFO:tasks.rados.rados.0.smithi071.stdout:639: finishing write tid 3 to smithi07183189-32 2023-12-15T22:59:17.055 INFO:tasks.rados.rados.0.smithi071.stdout:639: finishing write tid 4 to smithi07183189-32 2023-12-15T22:59:17.055 INFO:tasks.rados.rados.0.smithi071.stdout:639: finishing write tid 5 to smithi07183189-32 2023-12-15T22:59:17.055 INFO:tasks.rados.rados.0.smithi071.stdout:639: oid 32 updating version 0 to 45 2023-12-15T22:59:17.055 INFO:tasks.rados.rados.0.smithi071.stdout:639: oid 32 version 45 is already newer than 45 2023-12-15T22:59:17.056 INFO:tasks.rados.rados.0.smithi071.stdout:639: oid 32 version 45 is already newer than 42 2023-12-15T22:59:17.056 INFO:tasks.rados.rados.0.smithi071.stdout:639: oid 32 version 45 is already newer than 43 2023-12-15T22:59:17.056 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 32 v 45 (ObjNum 312 snap 81 seq_num 312) dirty exists 2023-12-15T22:59:17.056 INFO:tasks.rados.rados.0.smithi071.stdout:639: left oid 32 (ObjNum 312 snap 81 seq_num 312) 2023-12-15T22:59:17.071 INFO:tasks.rados.rados.0.smithi071.stdout:639: done (2 left) 2023-12-15T22:59:17.071 INFO:tasks.rados.rados.0.smithi071.stdout:641: done (1 left) 2023-12-15T22:59:17.072 INFO:tasks.rados.rados.0.smithi071.stdout:642: done (0 left) 2023-12-15T22:59:17.072 INFO:tasks.rados.rados.0.smithi071.stdout:643: snap_remove snap 75 2023-12-15T22:59:17.262 INFO:tasks.rados.rados.0.smithi071.stdout:643: done (0 left) 2023-12-15T22:59:17.262 INFO:tasks.rados.rados.0.smithi071.stdout:644: snap_remove snap 73 2023-12-15T22:59:17.842 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:59:17.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:17.946+0000 7fce93d08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:17.950 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:17.949+0000 7fce93d08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:18.264 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T22:59:18.264 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2023-12-15T22:59:18.267 INFO:tasks.rados.rados.0.smithi071.stdout:644: done (0 left) 2023-12-15T22:59:18.267 INFO:tasks.rados.rados.0.smithi071.stdout:645: read oid 39 snap -1 2023-12-15T22:59:18.267 INFO:tasks.rados.rados.0.smithi071.stdout:645: expect deleted 2023-12-15T22:59:18.267 INFO:tasks.rados.rados.0.smithi071.stdout:646: write oid 45 current snap is 81 2023-12-15T22:59:18.268 INFO:tasks.rados.rados.0.smithi071.stdout:646: seq_num 315 ranges {494205=473908,1433122=407102,2384125=462684} 2023-12-15T22:59:18.272 INFO:tasks.rados.rados.0.smithi071.stdout:646: writing smithi07183189-45 from 494205 to 968113 tid 1 2023-12-15T22:59:18.277 INFO:tasks.rados.rados.0.smithi071.stdout:646: writing smithi07183189-45 from 1433122 to 1840224 tid 2 2023-12-15T22:59:18.284 INFO:tasks.rados.rados.0.smithi071.stdout:646: writing smithi07183189-45 from 2384125 to 2846809 tid 3 2023-12-15T22:59:18.284 INFO:tasks.rados.rados.0.smithi071.stdout:647: delete oid 27 current snap is 81 2023-12-15T22:59:18.368 INFO:tasks.rados.rados.0.smithi071.stdout:646: finishing write tid 1 to smithi07183189-45 2023-12-15T22:59:18.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:18.382+0000 7ffb038f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:18.394 INFO:tasks.rados.rados.0.smithi071.stdout:645: done (2 left) 2023-12-15T22:59:18.394 INFO:tasks.rados.rados.0.smithi071.stdout:647: done (1 left) 2023-12-15T22:59:18.395 INFO:tasks.rados.rados.0.smithi071.stdout:648: snap_remove snap 79 2023-12-15T22:59:18.395 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:18.394+0000 7ffb038f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:19.291 INFO:teuthology.orchestra.run.smithi071.stderr:reweighted osd.0 to 1 (10000) 2023-12-15T22:59:19.298 INFO:tasks.rados.rados.0.smithi071.stdout:648: done (1 left) 2023-12-15T22:59:19.299 INFO:tasks.rados.rados.0.smithi071.stdout:649: write oid 34 current snap is 81 2023-12-15T22:59:19.299 INFO:tasks.rados.rados.0.smithi071.stdout:646: finishing write tid 2 to smithi07183189-45 2023-12-15T22:59:19.299 INFO:tasks.rados.rados.0.smithi071.stdout:649: seq_num 317 ranges {527819=624406,1833733=410771,2722618=601581} 2023-12-15T22:59:19.305 INFO:tasks.rados.rados.0.smithi071.stdout:649: writing smithi07183189-34 from 527819 to 1152225 tid 1 2023-12-15T22:59:19.313 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2023-12-15T22:59:19.314 INFO:tasks.rados.rados.0.smithi071.stdout:649: writing smithi07183189-34 from 1833733 to 2244504 tid 2 2023-12-15T22:59:19.318 INFO:tasks.rados.rados.0.smithi071.stdout:649: writing smithi07183189-34 from 2722618 to 3324199 tid 3 2023-12-15T22:59:19.318 INFO:tasks.rados.rados.0.smithi071.stdout:650: delete oid 33 current snap is 81 2023-12-15T22:59:19.318 INFO:tasks.rados.rados.0.smithi071.stdout:646: finishing write tid 3 to smithi07183189-45 2023-12-15T22:59:19.318 INFO:tasks.rados.rados.0.smithi071.stdout:646: finishing write tid 4 to smithi07183189-45 2023-12-15T22:59:19.318 INFO:tasks.rados.rados.0.smithi071.stdout:646: finishing write tid 5 to smithi07183189-45 2023-12-15T22:59:19.318 INFO:tasks.rados.rados.0.smithi071.stdout:646: oid 45 updating version 0 to 439 2023-12-15T22:59:19.318 INFO:tasks.rados.rados.0.smithi071.stdout:646: oid 45 version 439 is already newer than 439 2023-12-15T22:59:19.318 INFO:tasks.rados.rados.0.smithi071.stdout:646: oid 45 version 439 is already newer than 436 2023-12-15T22:59:19.318 INFO:tasks.rados.rados.0.smithi071.stdout:646: oid 45 version 439 is already newer than 439 2023-12-15T22:59:19.318 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 439 (ObjNum 315 snap 81 seq_num 315) dirty exists 2023-12-15T22:59:19.318 INFO:tasks.rados.rados.0.smithi071.stdout:646: left oid 45 (ObjNum 315 snap 81 seq_num 315) 2023-12-15T22:59:19.382 INFO:tasks.rados.rados.0.smithi071.stdout:646: done (2 left) 2023-12-15T22:59:19.382 INFO:tasks.rados.rados.0.smithi071.stdout:650: done (1 left) 2023-12-15T22:59:19.382 INFO:tasks.rados.rados.0.smithi071.stdout:651: read oid 17 snap -1 2023-12-15T22:59:19.382 INFO:tasks.rados.rados.0.smithi071.stdout:651: expect (ObjNum 220 snap 49 seq_num 220) 2023-12-15T22:59:19.382 INFO:tasks.rados.rados.0.smithi071.stdout:652: delete oid 13 current snap is 81 2023-12-15T22:59:19.399 INFO:tasks.rados.rados.0.smithi071.stdout:649: finishing write tid 1 to smithi07183189-34 2023-12-15T22:59:19.452 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:19.446+0000 7f37be22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:19.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:19.455+0000 7f37be22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:19.459 INFO:tasks.rados.rados.0.smithi071.stdout:652: done (2 left) 2023-12-15T22:59:19.459 INFO:tasks.rados.rados.0.smithi071.stdout:653: read oid 12 snap -1 2023-12-15T22:59:19.459 INFO:tasks.rados.rados.0.smithi071.stdout:653: expect deleted 2023-12-15T22:59:19.459 INFO:tasks.rados.rados.0.smithi071.stdout:654: delete oid 13 current snap is 81 2023-12-15T22:59:19.474 INFO:tasks.rados.rados.0.smithi071.stdout:649: finishing write tid 2 to smithi07183189-34 2023-12-15T22:59:19.493 INFO:tasks.rados.rados.0.smithi071.stdout:653: done (3 left) 2023-12-15T22:59:19.494 INFO:tasks.rados.rados.0.smithi071.stdout:654: done (2 left) 2023-12-15T22:59:19.494 INFO:tasks.rados.rados.0.smithi071.stdout:655: read oid 24 snap -1 2023-12-15T22:59:19.494 INFO:tasks.rados.rados.0.smithi071.stdout:655: expect (ObjNum 296 snap 80 seq_num 296) 2023-12-15T22:59:19.494 INFO:tasks.rados.rados.0.smithi071.stdout:656: snap_remove snap 63 2023-12-15T22:59:20.019 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:59:20.124 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:20.122+0000 7fcf76d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:20.129 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:20.127+0000 7fcf76d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:20.295 INFO:teuthology.orchestra.run.smithi071.stderr:reweighted osd.1 to 1 (10000) 2023-12-15T22:59:20.298 INFO:tasks.rados.rados.0.smithi071.stdout:656: done (3 left) 2023-12-15T22:59:20.298 INFO:tasks.rados.rados.0.smithi071.stdout:657: read oid 31 snap -1 2023-12-15T22:59:20.298 INFO:tasks.rados.rados.0.smithi071.stdout:657: expect deleted 2023-12-15T22:59:20.298 INFO:tasks.rados.rados.0.smithi071.stdout:658: write oid 9 current snap is 81 2023-12-15T22:59:20.299 INFO:tasks.rados.rados.0.smithi071.stdout:651: expect (ObjNum 220 snap 49 seq_num 220) 2023-12-15T22:59:20.311 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2023-12-15T22:59:20.427 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:20.421+0000 7fb95aa6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:20.430 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:20.430+0000 7fb95aa6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:20.590 INFO:tasks.rados.rados.0.smithi071.stdout:649: finishing write tid 3 to smithi07183189-34 2023-12-15T22:59:20.590 INFO:tasks.rados.rados.0.smithi071.stdout:658: seq_num 321 ranges {595047=641755,1667661=418109,2548663=763022} 2023-12-15T22:59:20.596 INFO:tasks.rados.rados.0.smithi071.stdout:658: writing smithi07183189-9 from 595047 to 1236802 tid 1 2023-12-15T22:59:20.601 INFO:tasks.rados.rados.0.smithi071.stdout:658: writing smithi07183189-9 from 1667661 to 2085770 tid 2 2023-12-15T22:59:20.608 INFO:tasks.rados.rados.0.smithi071.stdout:658: writing smithi07183189-9 from 2548663 to 3311685 tid 3 2023-12-15T22:59:20.608 INFO:tasks.rados.rados.0.smithi071.stdout:651: done (4 left) 2023-12-15T22:59:20.608 INFO:tasks.rados.rados.0.smithi071.stdout:659: read oid 14 snap -1 2023-12-15T22:59:20.608 INFO:tasks.rados.rados.0.smithi071.stdout:659: expect (ObjNum 232 snap 55 seq_num 232) 2023-12-15T22:59:20.608 INFO:tasks.rados.rados.0.smithi071.stdout:660: read oid 27 snap -1 2023-12-15T22:59:20.608 INFO:tasks.rados.rados.0.smithi071.stdout:660: expect deleted 2023-12-15T22:59:20.609 INFO:tasks.rados.rados.0.smithi071.stdout:661: write (excl) oid 16 current snap is 81 2023-12-15T22:59:20.609 INFO:tasks.rados.rados.0.smithi071.stdout:655: expect (ObjNum 296 snap 80 seq_num 296) 2023-12-15T22:59:20.842 INFO:tasks.rados.rados.0.smithi071.stdout:649: finishing write tid 4 to smithi07183189-34 2023-12-15T22:59:20.842 INFO:tasks.rados.rados.0.smithi071.stdout:649: finishing write tid 5 to smithi07183189-34 2023-12-15T22:59:20.842 INFO:tasks.rados.rados.0.smithi071.stdout:649: oid 34 updating version 0 to 50 2023-12-15T22:59:20.842 INFO:tasks.rados.rados.0.smithi071.stdout:649: oid 34 updating version 50 to 51 2023-12-15T22:59:20.842 INFO:tasks.rados.rados.0.smithi071.stdout:649: oid 34 version 51 is already newer than 48 2023-12-15T22:59:20.842 INFO:tasks.rados.rados.0.smithi071.stdout:649: oid 34 version 51 is already newer than 49 2023-12-15T22:59:20.842 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 51 (ObjNum 317 snap 81 seq_num 317) dirty exists 2023-12-15T22:59:20.843 INFO:tasks.rados.rados.0.smithi071.stdout:649: left oid 34 (ObjNum 317 snap 81 seq_num 317) 2023-12-15T22:59:20.843 INFO:tasks.rados.rados.0.smithi071.stdout:661: seq_num 322 ranges {611854=717004,2126143=561767,3259733=152121} 2023-12-15T22:59:20.848 INFO:tasks.rados.rados.0.smithi071.stdout:661: writing smithi07183189-16 from 611854 to 1328858 tid 1 2023-12-15T22:59:20.856 INFO:tasks.rados.rados.0.smithi071.stdout:661: writing smithi07183189-16 from 2126143 to 2687910 tid 2 2023-12-15T22:59:20.861 INFO:tasks.rados.rados.0.smithi071.stdout:661: writing smithi07183189-16 from 3259733 to 3411854 tid 3 2023-12-15T22:59:20.861 INFO:tasks.rados.rados.0.smithi071.stdout:649: done (6 left) 2023-12-15T22:59:20.861 INFO:tasks.rados.rados.0.smithi071.stdout:655: done (5 left) 2023-12-15T22:59:20.861 INFO:tasks.rados.rados.0.smithi071.stdout:657: done (4 left) 2023-12-15T22:59:20.861 INFO:tasks.rados.rados.0.smithi071.stdout:662: read oid 18 snap -1 2023-12-15T22:59:20.861 INFO:tasks.rados.rados.0.smithi071.stdout:662: expect deleted 2023-12-15T22:59:20.861 INFO:tasks.rados.rados.0.smithi071.stdout:663: write (excl) oid 11 current snap is 81 2023-12-15T22:59:20.861 INFO:tasks.rados.rados.0.smithi071.stdout:663: seq_num 323 ranges {628661=792253,2184625=421778,3403509=25152} 2023-12-15T22:59:20.868 INFO:tasks.rados.rados.0.smithi071.stdout:663: writing smithi07183189-11 from 628661 to 1420914 tid 1 2023-12-15T22:59:20.875 INFO:tasks.rados.rados.0.smithi071.stdout:663: writing smithi07183189-11 from 2184625 to 2606403 tid 2 2023-12-15T22:59:20.880 INFO:tasks.rados.rados.0.smithi071.stdout:663: writing smithi07183189-11 from 3403509 to 3428661 tid 3 2023-12-15T22:59:20.880 INFO:tasks.rados.rados.0.smithi071.stdout:664: snap_remove snap 76 2023-12-15T22:59:21.300 INFO:teuthology.orchestra.run.smithi071.stderr:reweighted osd.2 to 1 (10000) 2023-12-15T22:59:21.303 INFO:tasks.rados.rados.0.smithi071.stdout:664: done (6 left) 2023-12-15T22:59:21.303 INFO:tasks.rados.rados.0.smithi071.stdout:665: delete oid 44 current snap is 81 2023-12-15T22:59:21.303 INFO:tasks.rados.rados.0.smithi071.stdout:659: expect (ObjNum 232 snap 55 seq_num 232) 2023-12-15T22:59:21.313 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2023-12-15T22:59:21.423 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:21.418+0000 7f0b0206d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:21.426 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:21.426+0000 7f0b0206d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:21.686 INFO:tasks.rados.rados.0.smithi071.stdout:658: finishing write tid 1 to smithi07183189-9 2023-12-15T22:59:21.686 INFO:tasks.rados.rados.0.smithi071.stdout:658: finishing write tid 2 to smithi07183189-9 2023-12-15T22:59:21.686 INFO:tasks.rados.rados.0.smithi071.stdout:658: finishing write tid 3 to smithi07183189-9 2023-12-15T22:59:21.686 INFO:tasks.rados.rados.0.smithi071.stdout:658: finishing write tid 4 to smithi07183189-9 2023-12-15T22:59:21.686 INFO:tasks.rados.rados.0.smithi071.stdout:658: finishing write tid 5 to smithi07183189-9 2023-12-15T22:59:21.686 INFO:tasks.rados.rados.0.smithi071.stdout:658: oid 9 updating version 0 to 193 2023-12-15T22:59:21.686 INFO:tasks.rados.rados.0.smithi071.stdout:658: oid 9 updating version 193 to 199 2023-12-15T22:59:21.686 INFO:tasks.rados.rados.0.smithi071.stdout:658: oid 9 version 199 is already newer than 193 2023-12-15T22:59:21.687 INFO:tasks.rados.rados.0.smithi071.stdout:658: oid 9 version 199 is already newer than 198 2023-12-15T22:59:21.687 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 199 (ObjNum 321 snap 81 seq_num 321) dirty exists 2023-12-15T22:59:21.687 INFO:tasks.rados.rados.0.smithi071.stdout:658: left oid 9 (ObjNum 321 snap 81 seq_num 321) 2023-12-15T22:59:21.687 INFO:tasks.rados.rados.0.smithi071.stdout:658: done (6 left) 2023-12-15T22:59:21.687 INFO:tasks.rados.rados.0.smithi071.stdout:659: done (5 left) 2023-12-15T22:59:21.687 INFO:tasks.rados.rados.0.smithi071.stdout:660: done (4 left) 2023-12-15T22:59:21.687 INFO:tasks.rados.rados.0.smithi071.stdout:665: done (3 left) 2023-12-15T22:59:21.687 INFO:tasks.rados.rados.0.smithi071.stdout:666: delete oid 47 current snap is 81 2023-12-15T22:59:21.687 INFO:tasks.rados.rados.0.smithi071.stdout:661: finishing write tid 1 to smithi07183189-16 2023-12-15T22:59:21.688 INFO:tasks.rados.rados.0.smithi071.stdout:663: finishing write tid 1 to smithi07183189-11 2023-12-15T22:59:21.688 INFO:tasks.rados.rados.0.smithi071.stdout:661: finishing write tid 2 to smithi07183189-16 2023-12-15T22:59:21.688 INFO:tasks.rados.rados.0.smithi071.stdout:663: finishing write tid 2 to smithi07183189-11 2023-12-15T22:59:21.688 INFO:tasks.rados.rados.0.smithi071.stdout:663: finishing write tid 3 to smithi07183189-11 2023-12-15T22:59:21.688 INFO:tasks.rados.rados.0.smithi071.stdout:661: finishing write tid 3 to smithi07183189-16 2023-12-15T22:59:21.688 INFO:tasks.rados.rados.0.smithi071.stdout:663: finishing write tid 4 to smithi07183189-11 2023-12-15T22:59:21.688 INFO:tasks.rados.rados.0.smithi071.stdout:663: finishing write tid 5 to smithi07183189-11 2023-12-15T22:59:21.688 INFO:tasks.rados.rados.0.smithi071.stdout:663: oid 11 updating version 0 to 140 2023-12-15T22:59:21.688 INFO:tasks.rados.rados.0.smithi071.stdout:663: oid 11 version 140 is already newer than 140 2023-12-15T22:59:21.688 INFO:tasks.rados.rados.0.smithi071.stdout:663: oid 11 version 140 is already newer than 138 2023-12-15T22:59:21.688 INFO:tasks.rados.rados.0.smithi071.stdout:663: oid 11 updating version 140 to 141 2023-12-15T22:59:21.688 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 11 v 141 (ObjNum 323 snap 81 seq_num 323) dirty exists 2023-12-15T22:59:21.688 INFO:tasks.rados.rados.0.smithi071.stdout:663: left oid 11 (ObjNum 323 snap 81 seq_num 323) 2023-12-15T22:59:21.688 INFO:tasks.rados.rados.0.smithi071.stdout:661: finishing write tid 4 to smithi07183189-16 2023-12-15T22:59:21.688 INFO:tasks.rados.rados.0.smithi071.stdout:661: finishing write tid 5 to smithi07183189-16 2023-12-15T22:59:21.688 INFO:tasks.rados.rados.0.smithi071.stdout:661: oid 16 updating version 0 to 230 2023-12-15T22:59:21.689 INFO:tasks.rados.rados.0.smithi071.stdout:661: oid 16 version 230 is already newer than 229 2023-12-15T22:59:21.689 INFO:tasks.rados.rados.0.smithi071.stdout:661: oid 16 version 230 is already newer than 228 2023-12-15T22:59:21.689 INFO:tasks.rados.rados.0.smithi071.stdout:661: oid 16 updating version 230 to 231 2023-12-15T22:59:21.689 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 16 v 231 (ObjNum 322 snap 81 seq_num 322) dirty exists 2023-12-15T22:59:21.689 INFO:tasks.rados.rados.0.smithi071.stdout:661: left oid 16 (ObjNum 322 snap 81 seq_num 322) 2023-12-15T22:59:21.708 INFO:tasks.rados.rados.0.smithi071.stdout:661: done (3 left) 2023-12-15T22:59:21.708 INFO:tasks.rados.rados.0.smithi071.stdout:662: done (2 left) 2023-12-15T22:59:21.708 INFO:tasks.rados.rados.0.smithi071.stdout:663: done (1 left) 2023-12-15T22:59:21.708 INFO:tasks.rados.rados.0.smithi071.stdout:666: done (0 left) 2023-12-15T22:59:21.708 INFO:tasks.rados.rados.0.smithi071.stdout:667: read oid 17 snap -1 2023-12-15T22:59:21.708 INFO:tasks.rados.rados.0.smithi071.stdout:667: expect (ObjNum 220 snap 49 seq_num 220) 2023-12-15T22:59:21.708 INFO:tasks.rados.rados.0.smithi071.stdout:668: read oid 6 snap -1 2023-12-15T22:59:21.708 INFO:tasks.rados.rados.0.smithi071.stdout:668: expect (ObjNum 300 snap 80 seq_num 300) 2023-12-15T22:59:21.708 INFO:tasks.rados.rados.0.smithi071.stdout:669: read oid 42 snap -1 2023-12-15T22:59:21.708 INFO:tasks.rados.rados.0.smithi071.stdout:669: expect deleted 2023-12-15T22:59:21.709 INFO:tasks.rados.rados.0.smithi071.stdout:670: delete oid 14 current snap is 81 2023-12-15T22:59:21.763 INFO:tasks.rados.rados.0.smithi071.stdout:667: expect (ObjNum 220 snap 49 seq_num 220) 2023-12-15T22:59:22.054 INFO:tasks.rados.rados.0.smithi071.stdout:668: expect (ObjNum 300 snap 80 seq_num 300) 2023-12-15T22:59:22.195 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:59:22.299 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:22.298+0000 7f442e26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:22.302 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:22.303+0000 7f442e26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:22.303 INFO:teuthology.orchestra.run.smithi071.stderr:reweighted osd.3 to 1 (10000) 2023-12-15T22:59:22.307 INFO:tasks.rados.rados.0.smithi071.stdout:667: done (3 left) 2023-12-15T22:59:22.307 INFO:tasks.rados.rados.0.smithi071.stdout:668: done (2 left) 2023-12-15T22:59:22.307 INFO:tasks.rados.rados.0.smithi071.stdout:669: done (1 left) 2023-12-15T22:59:22.307 INFO:tasks.rados.rados.0.smithi071.stdout:670: done (0 left) 2023-12-15T22:59:22.307 INFO:tasks.rados.rados.0.smithi071.stdout:671: read oid 2 snap 80 2023-12-15T22:59:22.307 INFO:tasks.rados.rados.0.smithi071.stdout:671: expect (ObjNum 286 snap 78 seq_num 286) 2023-12-15T22:59:22.307 INFO:tasks.rados.rados.0.smithi071.stdout:672: delete oid 3 current snap is 81 2023-12-15T22:59:22.319 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2023-12-15T22:59:22.319 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T22:59:22.329 INFO:tasks.rados.rados.0.smithi071.stdout:671: expect (ObjNum 286 snap 78 seq_num 286) 2023-12-15T22:59:22.429 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:22.426+0000 7fe333b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:22.432 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:22.432+0000 7fe333b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:22.498 INFO:tasks.rados.rados.0.smithi071.stdout:671: done (1 left) 2023-12-15T22:59:22.498 INFO:tasks.rados.rados.0.smithi071.stdout:672: done (0 left) 2023-12-15T22:59:22.498 INFO:tasks.rados.rados.0.smithi071.stdout:673: read oid 32 snap -1 2023-12-15T22:59:22.498 INFO:tasks.rados.rados.0.smithi071.stdout:673: expect (ObjNum 312 snap 81 seq_num 312) 2023-12-15T22:59:22.498 INFO:tasks.rados.rados.0.smithi071.stdout:674: delete oid 7 current snap is 81 2023-12-15T22:59:22.507 INFO:tasks.rados.rados.0.smithi071.stdout:674: done (1 left) 2023-12-15T22:59:22.508 INFO:tasks.rados.rados.0.smithi071.stdout:675: snap_remove snap 51 2023-12-15T22:59:22.615 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:59:22.616 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-12-15T22:59:22.627 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":255,"stamp":"2023-12-15T22:59:22.544993+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92760616,"num_objects":229,"num_object_clones":164,"num_object_copies":687,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":32,"num_read":1389,"num_read_kb":945530,"num_write":1192,"num_write_kb":308265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2285,"ondisk_log_size":2285,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":546860,"kb_used_data":422924,"kb_used_omap":0,"kb_used_meta":123904,"kb_avail":376940500,"statfs":{"total":386547056640,"available":385987072000,"internally_reserved":0,"allocated":433074176,"data_stored":496796526,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":126877696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8201948,"num_objects":9,"num_object_clones":9,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":38,"num_read_kb":37372,"num_write":24,"num_write_kb":7161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000598"},"pg_stats":[{"pgid":"3.f","version":"172'57","reported_seq":388,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-12-15T22:59:19.309964+0000","last_change":"2023-12-15T22:59:19.359087+0000","last_active":"2023-12-15T22:59:19.309964+0000","last_peered":"2023-12-15T22:59:19.309964+0000","last_clean":"2023-12-15T22:59:19.309964+0000","last_became_active":"2023-12-15T22:56:39.503797+0000","last_became_peered":"2023-12-15T22:56:39.503797+0000","last_unstale":"2023-12-15T22:59:19.309964+0000","last_undegraded":"2023-12-15T22:59:19.309964+0000","last_fullsized":"2023-12-15T22:59:19.309964+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3243784,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":56,"num_read_kb":39856,"num_write":30,"num_write_kb":9157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":317,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-12-15T22:59:19.321931+0000","last_change":"2023-12-15T22:59:19.338878+0000","last_active":"2023-12-15T22:59:19.321931+0000","last_peered":"2023-12-15T22:59:19.321931+0000","last_clean":"2023-12-15T22:59:19.321931+0000","last_became_active":"2023-12-15T22:56:39.552856+0000","last_became_peered":"2023-12-15T22:56:39.552856+0000","last_unstale":"2023-12-15T22:59:19.321931+0000","last_undegraded":"2023-12-15T22:59:19.321931+0000","last_fullsized":"2023-12-15T22:59:19.321931+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"172'222","reported_seq":514,"reported_epoch":172,"state":"active+clean+snaptrim","last_fresh":"2023-12-15T22:59:19.488171+0000","last_change":"2023-12-15T22:59:19.419137+0000","last_active":"2023-12-15T22:59:19.488171+0000","last_peered":"2023-12-15T22:59:19.488171+0000","last_clean":"2023-12-15T22:59:19.488171+0000","last_became_active":"2023-12-15T22:56:39.495490+0000","last_became_peered":"2023-12-15T22:56:39.495490+0000","last_unstale":"2023-12-15T22:59:19.488171+0000","last_undegraded":"2023-12-15T22:59:19.488171+0000","last_fullsized":"2023-12-15T22:59:19.488171+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13059072,"num_objects":25,"num_object_clones":16,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":191,"num_read_kb":170372,"num_write":127,"num_write_kb":36642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"172'190","reported_seq":478,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-12-15T22:59:19.308519+0000","last_change":"2023-12-15T22:59:19.405989+0000","last_active":"2023-12-15T22:59:19.308519+0000","last_peered":"2023-12-15T22:59:19.308519+0000","last_clean":"2023-12-15T22:59:19.308519+0000","last_became_active":"2023-12-15T22:56:39.579234+0000","last_became_peered":"2023-12-15T22:56:39.579234+0000","last_unstale":"2023-12-15T22:59:19.308519+0000","last_undegraded":"2023-12-15T22:59:19.308519+0000","last_fullsized":"2023-12-15T22:59:19.308519+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5126201,"num_objects":19,"num_object_clones":16,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":123,"num_read_kb":90328,"num_write":99,"num_write_kb":27559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"172'105","reported_seq":393,"reported_epoch":173,"state":"active+clean+snaptrim","last_fresh":"2023-12-15T22:59:20.309783+0000","last_change":"2023-12-15T22:59:19.430299+0000","last_active":"2023-12-15T22:59:20.309783+0000","last_peered":"2023-12-15T22:59:20.309783+0000","last_clean":"2023-12-15T22:59:20.309783+0000","last_became_active":"2023-12-15T22:56:39.572176+0000","last_became_peered":"2023-12-15T22:56:39.572176+0000","last_unstale":"2023-12-15T22:59:20.309783+0000","last_undegraded":"2023-12-15T22:59:20.309783+0000","last_fullsized":"2023-12-15T22:59:20.309783+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2756381,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":53,"num_read_kb":29044,"num_write":45,"num_write_kb":13074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"172'116","reported_seq":400,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-12-15T22:59:19.311384+0000","last_change":"2023-12-15T22:59:19.364654+0000","last_active":"2023-12-15T22:59:19.311384+0000","last_peered":"2023-12-15T22:59:19.311384+0000","last_clean":"2023-12-15T22:59:19.311384+0000","last_became_active":"2023-12-15T22:56:39.516361+0000","last_became_peered":"2023-12-15T22:56:39.516361+0000","last_unstale":"2023-12-15T22:59:19.311384+0000","last_undegraded":"2023-12-15T22:59:19.311384+0000","last_fullsized":"2023-12-15T22:59:19.311384+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2806802,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":44,"num_read_kb":31597,"num_write":51,"num_write_kb":14458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"172'455","reported_seq":602,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-12-15T22:59:19.382629+0000","last_change":"2023-12-15T22:59:19.512726+0000","last_active":"2023-12-15T22:59:19.382629+0000","last_peered":"2023-12-15T22:59:19.382629+0000","last_clean":"2023-12-15T22:59:19.382629+0000","last_became_active":"2023-12-15T22:56:39.519748+0000","last_became_peered":"2023-12-15T22:56:39.519748+0000","last_unstale":"2023-12-15T22:59:19.382629+0000","last_undegraded":"2023-12-15T22:59:19.382629+0000","last_fullsized":"2023-12-15T22:59:19.382629+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":455,"log_dups_size":0,"ondisk_log_size":455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12151593,"num_objects":47,"num_object_clones":37,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":190,"num_read_kb":113288,"num_write":195,"num_write_kb":56172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"169'98","reported_seq":402,"reported_epoch":172,"state":"active+clean+snaptrim","last_fresh":"2023-12-15T22:59:19.430176+0000","last_change":"2023-12-15T22:59:19.430176+0000","last_active":"2023-12-15T22:59:19.430176+0000","last_peered":"2023-12-15T22:59:19.430176+0000","last_clean":"2023-12-15T22:59:19.430176+0000","last_became_active":"2023-12-15T22:56:39.543179+0000","last_became_peered":"2023-12-15T22:56:39.543179+0000","last_unstale":"2023-12-15T22:59:19.430176+0000","last_undegraded":"2023-12-15T22:59:19.430176+0000","last_fullsized":"2023-12-15T22:59:19.430176+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8790061,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":71707,"num_write":62,"num_write_kb":16612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":388,"reported_epoch":173,"state":"active+clean","last_fresh":"2023-12-15T22:59:20.313196+0000","last_change":"2023-12-15T22:56:32.607017+0000","last_active":"2023-12-15T22:59:20.313196+0000","last_peered":"2023-12-15T22:59:20.313196+0000","last_clean":"2023-12-15T22:59:20.313196+0000","last_became_active":"2023-12-15T22:56:32.602384+0000","last_became_peered":"2023-12-15T22:56:32.602384+0000","last_unstale":"2023-12-15T22:59:20.313196+0000","last_undegraded":"2023-12-15T22:59:20.313196+0000","last_fullsized":"2023-12-15T22:59:20.313196+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_clean_scrub_stamp":"2023-12-15T22:56:31.525572+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":245,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-12-15T22:59:19.428208+0000","last_change":"2023-12-15T22:56:33.627481+0000","last_active":"2023-12-15T22:59:19.428208+0000","last_peered":"2023-12-15T22:59:19.428208+0000","last_clean":"2023-12-15T22:59:19.428208+0000","last_became_active":"2023-12-15T22:56:33.624689+0000","last_became_peered":"2023-12-15T22:56:33.624689+0000","last_unstale":"2023-12-15T22:59:19.428208+0000","last_undegraded":"2023-12-15T22:59:19.428208+0000","last_fullsized":"2023-12-15T22:59:19.428208+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"169'132","reported_seq":436,"reported_epoch":172,"state":"active+clean+snaptrim","last_fresh":"2023-12-15T22:59:19.420513+0000","last_change":"2023-12-15T22:59:19.420513+0000","last_active":"2023-12-15T22:59:19.420513+0000","last_peered":"2023-12-15T22:59:19.420513+0000","last_clean":"2023-12-15T22:59:19.420513+0000","last_became_active":"2023-12-15T22:56:39.529359+0000","last_became_peered":"2023-12-15T22:56:39.529359+0000","last_unstale":"2023-12-15T22:59:19.420513+0000","last_undegraded":"2023-12-15T22:59:19.420513+0000","last_fullsized":"2023-12-15T22:59:19.420513+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8924550,"num_objects":14,"num_object_clones":8,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":109,"num_read_kb":81765,"num_write":79,"num_write_kb":23955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":267,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-12-15T22:59:19.314447+0000","last_change":"2023-12-15T22:56:33.655797+0000","last_active":"2023-12-15T22:59:19.314447+0000","last_peered":"2023-12-15T22:59:19.314447+0000","last_clean":"2023-12-15T22:59:19.314447+0000","last_became_active":"2023-12-15T22:56:33.652885+0000","last_became_peered":"2023-12-15T22:56:33.652885+0000","last_unstale":"2023-12-15T22:59:19.314447+0000","last_undegraded":"2023-12-15T22:59:19.314447+0000","last_fullsized":"2023-12-15T22:59:19.314447+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"172'206","reported_seq":463,"reported_epoch":173,"state":"active+clean+snaptrim","last_fresh":"2023-12-15T22:59:20.310719+0000","last_change":"2023-12-15T22:59:19.431375+0000","last_active":"2023-12-15T22:59:20.310719+0000","last_peered":"2023-12-15T22:59:20.310719+0000","last_clean":"2023-12-15T22:59:20.310719+0000","last_became_active":"2023-12-15T22:56:39.543509+0000","last_became_peered":"2023-12-15T22:56:39.543509+0000","last_unstale":"2023-12-15T22:59:20.310719+0000","last_undegraded":"2023-12-15T22:59:20.310719+0000","last_fullsized":"2023-12-15T22:59:20.310719+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":206,"log_dups_size":0,"ondisk_log_size":206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5949711,"num_objects":21,"num_object_clones":15,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":124,"num_read_kb":86578,"num_write":85,"num_write_kb":25517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":245,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-12-15T22:59:19.424146+0000","last_change":"2023-12-15T22:56:33.600485+0000","last_active":"2023-12-15T22:59:19.424146+0000","last_peered":"2023-12-15T22:59:19.424146+0000","last_clean":"2023-12-15T22:59:19.424146+0000","last_became_active":"2023-12-15T22:56:33.597320+0000","last_became_peered":"2023-12-15T22:56:33.597320+0000","last_unstale":"2023-12-15T22:59:19.424146+0000","last_undegraded":"2023-12-15T22:59:19.424146+0000","last_fullsized":"2023-12-15T22:59:19.424146+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"172'83","reported_seq":375,"reported_epoch":173,"state":"active+clean+snaptrim","last_fresh":"2023-12-15T22:59:20.312423+0000","last_change":"2023-12-15T22:59:19.434909+0000","last_active":"2023-12-15T22:59:20.312423+0000","last_peered":"2023-12-15T22:59:20.312423+0000","last_clean":"2023-12-15T22:59:20.312423+0000","last_became_active":"2023-12-15T22:56:39.560980+0000","last_became_peered":"2023-12-15T22:56:39.560980+0000","last_unstale":"2023-12-15T22:59:20.312423+0000","last_undegraded":"2023-12-15T22:59:20.312423+0000","last_fullsized":"2023-12-15T22:59:20.312423+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6605184,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":29100,"num_write":35,"num_write_kb":10458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":267,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-12-15T22:59:19.309612+0000","last_change":"2023-12-15T22:56:33.648541+0000","last_active":"2023-12-15T22:59:19.309612+0000","last_peered":"2023-12-15T22:59:19.309612+0000","last_clean":"2023-12-15T22:59:19.309612+0000","last_became_active":"2023-12-15T22:56:33.643681+0000","last_became_peered":"2023-12-15T22:56:33.643681+0000","last_unstale":"2023-12-15T22:59:19.309612+0000","last_undegraded":"2023-12-15T22:59:19.309612+0000","last_fullsized":"2023-12-15T22:59:19.309612+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"172'159","reported_seq":425,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-12-15T22:59:19.336459+0000","last_change":"2023-12-15T22:59:19.415769+0000","last_active":"2023-12-15T22:59:19.336459+0000","last_peered":"2023-12-15T22:59:19.336459+0000","last_clean":"2023-12-15T22:59:19.336459+0000","last_became_active":"2023-12-15T22:56:39.506510+0000","last_became_peered":"2023-12-15T22:56:39.506510+0000","last_unstale":"2023-12-15T22:59:19.336459+0000","last_undegraded":"2023-12-15T22:59:19.336459+0000","last_fullsized":"2023-12-15T22:59:19.336459+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4773254,"num_objects":17,"num_object_clones":12,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":94,"num_read_kb":66513,"num_write":71,"num_write_kb":20602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"172'80","reported_seq":351,"reported_epoch":172,"state":"active+clean+snaptrim","last_fresh":"2023-12-15T22:59:19.527744+0000","last_change":"2023-12-15T22:59:19.425477+0000","last_active":"2023-12-15T22:59:19.527744+0000","last_peered":"2023-12-15T22:59:19.527744+0000","last_clean":"2023-12-15T22:59:19.527744+0000","last_became_active":"2023-12-15T22:56:39.511159+0000","last_became_peered":"2023-12-15T22:56:39.511159+0000","last_unstale":"2023-12-15T22:59:19.527744+0000","last_undegraded":"2023-12-15T22:59:19.527744+0000","last_fullsized":"2023-12-15T22:59:19.527744+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2974872,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":20,"num_read_kb":7340,"num_write":37,"num_write_kb":11102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":247,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-12-15T22:59:19.332983+0000","last_change":"2023-12-15T22:56:33.633890+0000","last_active":"2023-12-15T22:59:19.332983+0000","last_peered":"2023-12-15T22:59:19.332983+0000","last_clean":"2023-12-15T22:59:19.332983+0000","last_became_active":"2023-12-15T22:56:33.631134+0000","last_became_peered":"2023-12-15T22:56:33.631134+0000","last_unstale":"2023-12-15T22:59:19.332983+0000","last_undegraded":"2023-12-15T22:59:19.332983+0000","last_fullsized":"2023-12-15T22:59:19.332983+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"172'95","reported_seq":376,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-12-15T22:59:19.339304+0000","last_change":"2023-12-15T22:59:19.407340+0000","last_active":"2023-12-15T22:59:19.339304+0000","last_peered":"2023-12-15T22:59:19.339304+0000","last_clean":"2023-12-15T22:59:19.339304+0000","last_became_active":"2023-12-15T22:56:39.532730+0000","last_became_peered":"2023-12-15T22:56:39.532730+0000","last_unstale":"2023-12-15T22:59:19.339304+0000","last_undegraded":"2023-12-15T22:59:19.339304+0000","last_fullsized":"2023-12-15T22:59:19.339304+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5865709,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":47,"num_read_kb":38482,"num_write":44,"num_write_kb":13247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":259,"reported_epoch":173,"state":"active+clean","last_fresh":"2023-12-15T22:59:20.315512+0000","last_change":"2023-12-15T22:56:33.655736+0000","last_active":"2023-12-15T22:59:20.315512+0000","last_peered":"2023-12-15T22:59:20.315512+0000","last_clean":"2023-12-15T22:59:20.315512+0000","last_became_active":"2023-12-15T22:56:33.652917+0000","last_became_peered":"2023-12-15T22:56:33.652917+0000","last_unstale":"2023-12-15T22:59:20.315512+0000","last_undegraded":"2023-12-15T22:59:20.315512+0000","last_fullsized":"2023-12-15T22:59:20.315512+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"172'139","reported_seq":434,"reported_epoch":173,"state":"active+clean+snaptrim","last_fresh":"2023-12-15T22:59:20.316231+0000","last_change":"2023-12-15T22:59:19.427821+0000","last_active":"2023-12-15T22:59:20.316231+0000","last_peered":"2023-12-15T22:59:20.316231+0000","last_clean":"2023-12-15T22:59:20.316231+0000","last_became_active":"2023-12-15T22:56:39.557252+0000","last_became_peered":"2023-12-15T22:56:39.557252+0000","last_unstale":"2023-12-15T22:59:20.316231+0000","last_undegraded":"2023-12-15T22:59:20.316231+0000","last_fullsized":"2023-12-15T22:59:20.316231+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5815255,"num_objects":13,"num_object_clones":9,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":82,"num_read_kb":60078,"num_write":73,"num_write_kb":21141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":247,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-12-15T22:59:19.326094+0000","last_change":"2023-12-15T22:56:33.632348+0000","last_active":"2023-12-15T22:59:19.326094+0000","last_peered":"2023-12-15T22:59:19.326094+0000","last_clean":"2023-12-15T22:59:19.326094+0000","last_became_active":"2023-12-15T22:56:33.629022+0000","last_became_peered":"2023-12-15T22:56:33.629022+0000","last_unstale":"2023-12-15T22:59:19.326094+0000","last_undegraded":"2023-12-15T22:59:19.326094+0000","last_fullsized":"2023-12-15T22:59:19.326094+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"172'51","reported_seq":351,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-12-15T22:59:19.508151+0000","last_change":"2023-12-15T22:59:19.339003+0000","last_active":"2023-12-15T22:59:19.508151+0000","last_peered":"2023-12-15T22:59:19.508151+0000","last_clean":"2023-12-15T22:59:19.508151+0000","last_became_active":"2023-12-15T22:56:39.535145+0000","last_became_peered":"2023-12-15T22:56:39.535145+0000","last_unstale":"2023-12-15T22:59:19.508151+0000","last_undegraded":"2023-12-15T22:59:19.508151+0000","last_fullsized":"2023-12-15T22:59:19.508151+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3327819,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":29391,"num_write":22,"num_write_kb":6675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":245,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-12-15T22:59:19.422178+0000","last_change":"2023-12-15T22:56:33.627005+0000","last_active":"2023-12-15T22:59:19.422178+0000","last_peered":"2023-12-15T22:59:19.422178+0000","last_clean":"2023-12-15T22:59:19.422178+0000","last_became_active":"2023-12-15T22:56:33.622181+0000","last_became_peered":"2023-12-15T22:56:33.622181+0000","last_unstale":"2023-12-15T22:59:19.422178+0000","last_undegraded":"2023-12-15T22:59:19.422178+0000","last_fullsized":"2023-12-15T22:59:19.422178+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":92170248,"num_objects":227,"num_object_clones":164,"num_object_copies":681,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":32,"num_read":1286,"num_read_kb":945439,"num_write":1055,"num_write_kb":306371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2188,"ondisk_log_size":2188,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705834,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":142440,"kb_used_data":111456,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94229400,"statfs":{"total":96636764160,"available":96490905600,"internally_reserved":0,"allocated":114130944,"data_stored":132373887,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705833,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":121596,"kb_used_data":90612,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94250244,"statfs":{"total":96636764160,"available":96512249856,"internally_reserved":0,"allocated":92786688,"data_stored":107550539,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705833,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":139200,"kb_used_data":108216,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94232640,"statfs":{"total":96636764160,"available":96494223360,"internally_reserved":0,"allocated":110813184,"data_stored":126659602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705834,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":143624,"kb_used_data":112640,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94228216,"statfs":{"total":96636764160,"available":96489693184,"internally_reserved":0,"allocated":115343360,"data_stored":130212498,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-15T22:59:22.629 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T22:59:22.776 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:22.772+0000 7f9e1219e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:22.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:22.778+0000 7f9e1219e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:22.950 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:59:22.951 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-12-15T22:59:22.963 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":255,"stamp":"2023-12-15T22:59:22.544993+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92760616,"num_objects":229,"num_object_clones":164,"num_object_copies":687,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":32,"num_read":1389,"num_read_kb":945530,"num_write":1192,"num_write_kb":308265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2285,"ondisk_log_size":2285,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":546860,"kb_used_data":422924,"kb_used_omap":0,"kb_used_meta":123904,"kb_avail":376940500,"statfs":{"total":386547056640,"available":385987072000,"internally_reserved":0,"allocated":433074176,"data_stored":496796526,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":126877696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8201948,"num_objects":9,"num_object_clones":9,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":38,"num_read_kb":37372,"num_write":24,"num_write_kb":7161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000598"},"pg_stats":[{"pgid":"3.f","version":"172'57","reported_seq":388,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-12-15T22:59:19.309964+0000","last_change":"2023-12-15T22:59:19.359087+0000","last_active":"2023-12-15T22:59:19.309964+0000","last_peered":"2023-12-15T22:59:19.309964+0000","last_clean":"2023-12-15T22:59:19.309964+0000","last_became_active":"2023-12-15T22:56:39.503797+0000","last_became_peered":"2023-12-15T22:56:39.503797+0000","last_unstale":"2023-12-15T22:59:19.309964+0000","last_undegraded":"2023-12-15T22:59:19.309964+0000","last_fullsized":"2023-12-15T22:59:19.309964+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3243784,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":56,"num_read_kb":39856,"num_write":30,"num_write_kb":9157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":317,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-12-15T22:59:19.321931+0000","last_change":"2023-12-15T22:59:19.338878+0000","last_active":"2023-12-15T22:59:19.321931+0000","last_peered":"2023-12-15T22:59:19.321931+0000","last_clean":"2023-12-15T22:59:19.321931+0000","last_became_active":"2023-12-15T22:56:39.552856+0000","last_became_peered":"2023-12-15T22:56:39.552856+0000","last_unstale":"2023-12-15T22:59:19.321931+0000","last_undegraded":"2023-12-15T22:59:19.321931+0000","last_fullsized":"2023-12-15T22:59:19.321931+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"172'222","reported_seq":514,"reported_epoch":172,"state":"active+clean+snaptrim","last_fresh":"2023-12-15T22:59:19.488171+0000","last_change":"2023-12-15T22:59:19.419137+0000","last_active":"2023-12-15T22:59:19.488171+0000","last_peered":"2023-12-15T22:59:19.488171+0000","last_clean":"2023-12-15T22:59:19.488171+0000","last_became_active":"2023-12-15T22:56:39.495490+0000","last_became_peered":"2023-12-15T22:56:39.495490+0000","last_unstale":"2023-12-15T22:59:19.488171+0000","last_undegraded":"2023-12-15T22:59:19.488171+0000","last_fullsized":"2023-12-15T22:59:19.488171+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13059072,"num_objects":25,"num_object_clones":16,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":191,"num_read_kb":170372,"num_write":127,"num_write_kb":36642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"172'190","reported_seq":478,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-12-15T22:59:19.308519+0000","last_change":"2023-12-15T22:59:19.405989+0000","last_active":"2023-12-15T22:59:19.308519+0000","last_peered":"2023-12-15T22:59:19.308519+0000","last_clean":"2023-12-15T22:59:19.308519+0000","last_became_active":"2023-12-15T22:56:39.579234+0000","last_became_peered":"2023-12-15T22:56:39.579234+0000","last_unstale":"2023-12-15T22:59:19.308519+0000","last_undegraded":"2023-12-15T22:59:19.308519+0000","last_fullsized":"2023-12-15T22:59:19.308519+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5126201,"num_objects":19,"num_object_clones":16,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":123,"num_read_kb":90328,"num_write":99,"num_write_kb":27559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"172'105","reported_seq":393,"reported_epoch":173,"state":"active+clean+snaptrim","last_fresh":"2023-12-15T22:59:20.309783+0000","last_change":"2023-12-15T22:59:19.430299+0000","last_active":"2023-12-15T22:59:20.309783+0000","last_peered":"2023-12-15T22:59:20.309783+0000","last_clean":"2023-12-15T22:59:20.309783+0000","last_became_active":"2023-12-15T22:56:39.572176+0000","last_became_peered":"2023-12-15T22:56:39.572176+0000","last_unstale":"2023-12-15T22:59:20.309783+0000","last_undegraded":"2023-12-15T22:59:20.309783+0000","last_fullsized":"2023-12-15T22:59:20.309783+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2756381,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":53,"num_read_kb":29044,"num_write":45,"num_write_kb":13074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"172'116","reported_seq":400,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-12-15T22:59:19.311384+0000","last_change":"2023-12-15T22:59:19.364654+0000","last_active":"2023-12-15T22:59:19.311384+0000","last_peered":"2023-12-15T22:59:19.311384+0000","last_clean":"2023-12-15T22:59:19.311384+0000","last_became_active":"2023-12-15T22:56:39.516361+0000","last_became_peered":"2023-12-15T22:56:39.516361+0000","last_unstale":"2023-12-15T22:59:19.311384+0000","last_undegraded":"2023-12-15T22:59:19.311384+0000","last_fullsized":"2023-12-15T22:59:19.311384+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2806802,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":44,"num_read_kb":31597,"num_write":51,"num_write_kb":14458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"172'455","reported_seq":602,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-12-15T22:59:19.382629+0000","last_change":"2023-12-15T22:59:19.512726+0000","last_active":"2023-12-15T22:59:19.382629+0000","last_peered":"2023-12-15T22:59:19.382629+0000","last_clean":"2023-12-15T22:59:19.382629+0000","last_became_active":"2023-12-15T22:56:39.519748+0000","last_became_peered":"2023-12-15T22:56:39.519748+0000","last_unstale":"2023-12-15T22:59:19.382629+0000","last_undegraded":"2023-12-15T22:59:19.382629+0000","last_fullsized":"2023-12-15T22:59:19.382629+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":455,"log_dups_size":0,"ondisk_log_size":455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12151593,"num_objects":47,"num_object_clones":37,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":190,"num_read_kb":113288,"num_write":195,"num_write_kb":56172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"169'98","reported_seq":402,"reported_epoch":172,"state":"active+clean+snaptrim","last_fresh":"2023-12-15T22:59:19.430176+0000","last_change":"2023-12-15T22:59:19.430176+0000","last_active":"2023-12-15T22:59:19.430176+0000","last_peered":"2023-12-15T22:59:19.430176+0000","last_clean":"2023-12-15T22:59:19.430176+0000","last_became_active":"2023-12-15T22:56:39.543179+0000","last_became_peered":"2023-12-15T22:56:39.543179+0000","last_unstale":"2023-12-15T22:59:19.430176+0000","last_undegraded":"2023-12-15T22:59:19.430176+0000","last_fullsized":"2023-12-15T22:59:19.430176+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8790061,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":71707,"num_write":62,"num_write_kb":16612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":388,"reported_epoch":173,"state":"active+clean","last_fresh":"2023-12-15T22:59:20.313196+0000","last_change":"2023-12-15T22:56:32.607017+0000","last_active":"2023-12-15T22:59:20.313196+0000","last_peered":"2023-12-15T22:59:20.313196+0000","last_clean":"2023-12-15T22:59:20.313196+0000","last_became_active":"2023-12-15T22:56:32.602384+0000","last_became_peered":"2023-12-15T22:56:32.602384+0000","last_unstale":"2023-12-15T22:59:20.313196+0000","last_undegraded":"2023-12-15T22:59:20.313196+0000","last_fullsized":"2023-12-15T22:59:20.313196+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_clean_scrub_stamp":"2023-12-15T22:56:31.525572+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":245,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-12-15T22:59:19.428208+0000","last_change":"2023-12-15T22:56:33.627481+0000","last_active":"2023-12-15T22:59:19.428208+0000","last_peered":"2023-12-15T22:59:19.428208+0000","last_clean":"2023-12-15T22:59:19.428208+0000","last_became_active":"2023-12-15T22:56:33.624689+0000","last_became_peered":"2023-12-15T22:56:33.624689+0000","last_unstale":"2023-12-15T22:59:19.428208+0000","last_undegraded":"2023-12-15T22:59:19.428208+0000","last_fullsized":"2023-12-15T22:59:19.428208+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"169'132","reported_seq":436,"reported_epoch":172,"state":"active+clean+snaptrim","last_fresh":"2023-12-15T22:59:19.420513+0000","last_change":"2023-12-15T22:59:19.420513+0000","last_active":"2023-12-15T22:59:19.420513+0000","last_peered":"2023-12-15T22:59:19.420513+0000","last_clean":"2023-12-15T22:59:19.420513+0000","last_became_active":"2023-12-15T22:56:39.529359+0000","last_became_peered":"2023-12-15T22:56:39.529359+0000","last_unstale":"2023-12-15T22:59:19.420513+0000","last_undegraded":"2023-12-15T22:59:19.420513+0000","last_fullsized":"2023-12-15T22:59:19.420513+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8924550,"num_objects":14,"num_object_clones":8,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":109,"num_read_kb":81765,"num_write":79,"num_write_kb":23955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":267,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-12-15T22:59:19.314447+0000","last_change":"2023-12-15T22:56:33.655797+0000","last_active":"2023-12-15T22:59:19.314447+0000","last_peered":"2023-12-15T22:59:19.314447+0000","last_clean":"2023-12-15T22:59:19.314447+0000","last_became_active":"2023-12-15T22:56:33.652885+0000","last_became_peered":"2023-12-15T22:56:33.652885+0000","last_unstale":"2023-12-15T22:59:19.314447+0000","last_undegraded":"2023-12-15T22:59:19.314447+0000","last_fullsized":"2023-12-15T22:59:19.314447+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"172'206","reported_seq":463,"reported_epoch":173,"state":"active+clean+snaptrim","last_fresh":"2023-12-15T22:59:20.310719+0000","last_change":"2023-12-15T22:59:19.431375+0000","last_active":"2023-12-15T22:59:20.310719+0000","last_peered":"2023-12-15T22:59:20.310719+0000","last_clean":"2023-12-15T22:59:20.310719+0000","last_became_active":"2023-12-15T22:56:39.543509+0000","last_became_peered":"2023-12-15T22:56:39.543509+0000","last_unstale":"2023-12-15T22:59:20.310719+0000","last_undegraded":"2023-12-15T22:59:20.310719+0000","last_fullsized":"2023-12-15T22:59:20.310719+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":206,"log_dups_size":0,"ondisk_log_size":206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5949711,"num_objects":21,"num_object_clones":15,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":124,"num_read_kb":86578,"num_write":85,"num_write_kb":25517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":245,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-12-15T22:59:19.424146+0000","last_change":"2023-12-15T22:56:33.600485+0000","last_active":"2023-12-15T22:59:19.424146+0000","last_peered":"2023-12-15T22:59:19.424146+0000","last_clean":"2023-12-15T22:59:19.424146+0000","last_became_active":"2023-12-15T22:56:33.597320+0000","last_became_peered":"2023-12-15T22:56:33.597320+0000","last_unstale":"2023-12-15T22:59:19.424146+0000","last_undegraded":"2023-12-15T22:59:19.424146+0000","last_fullsized":"2023-12-15T22:59:19.424146+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"172'83","reported_seq":375,"reported_epoch":173,"state":"active+clean+snaptrim","last_fresh":"2023-12-15T22:59:20.312423+0000","last_change":"2023-12-15T22:59:19.434909+0000","last_active":"2023-12-15T22:59:20.312423+0000","last_peered":"2023-12-15T22:59:20.312423+0000","last_clean":"2023-12-15T22:59:20.312423+0000","last_became_active":"2023-12-15T22:56:39.560980+0000","last_became_peered":"2023-12-15T22:56:39.560980+0000","last_unstale":"2023-12-15T22:59:20.312423+0000","last_undegraded":"2023-12-15T22:59:20.312423+0000","last_fullsized":"2023-12-15T22:59:20.312423+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6605184,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":29100,"num_write":35,"num_write_kb":10458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":267,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-12-15T22:59:19.309612+0000","last_change":"2023-12-15T22:56:33.648541+0000","last_active":"2023-12-15T22:59:19.309612+0000","last_peered":"2023-12-15T22:59:19.309612+0000","last_clean":"2023-12-15T22:59:19.309612+0000","last_became_active":"2023-12-15T22:56:33.643681+0000","last_became_peered":"2023-12-15T22:56:33.643681+0000","last_unstale":"2023-12-15T22:59:19.309612+0000","last_undegraded":"2023-12-15T22:59:19.309612+0000","last_fullsized":"2023-12-15T22:59:19.309612+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"172'159","reported_seq":425,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-12-15T22:59:19.336459+0000","last_change":"2023-12-15T22:59:19.415769+0000","last_active":"2023-12-15T22:59:19.336459+0000","last_peered":"2023-12-15T22:59:19.336459+0000","last_clean":"2023-12-15T22:59:19.336459+0000","last_became_active":"2023-12-15T22:56:39.506510+0000","last_became_peered":"2023-12-15T22:56:39.506510+0000","last_unstale":"2023-12-15T22:59:19.336459+0000","last_undegraded":"2023-12-15T22:59:19.336459+0000","last_fullsized":"2023-12-15T22:59:19.336459+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4773254,"num_objects":17,"num_object_clones":12,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":94,"num_read_kb":66513,"num_write":71,"num_write_kb":20602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"172'80","reported_seq":351,"reported_epoch":172,"state":"active+clean+snaptrim","last_fresh":"2023-12-15T22:59:19.527744+0000","last_change":"2023-12-15T22:59:19.425477+0000","last_active":"2023-12-15T22:59:19.527744+0000","last_peered":"2023-12-15T22:59:19.527744+0000","last_clean":"2023-12-15T22:59:19.527744+0000","last_became_active":"2023-12-15T22:56:39.511159+0000","last_became_peered":"2023-12-15T22:56:39.511159+0000","last_unstale":"2023-12-15T22:59:19.527744+0000","last_undegraded":"2023-12-15T22:59:19.527744+0000","last_fullsized":"2023-12-15T22:59:19.527744+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2974872,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":20,"num_read_kb":7340,"num_write":37,"num_write_kb":11102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":247,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-12-15T22:59:19.332983+0000","last_change":"2023-12-15T22:56:33.633890+0000","last_active":"2023-12-15T22:59:19.332983+0000","last_peered":"2023-12-15T22:59:19.332983+0000","last_clean":"2023-12-15T22:59:19.332983+0000","last_became_active":"2023-12-15T22:56:33.631134+0000","last_became_peered":"2023-12-15T22:56:33.631134+0000","last_unstale":"2023-12-15T22:59:19.332983+0000","last_undegraded":"2023-12-15T22:59:19.332983+0000","last_fullsized":"2023-12-15T22:59:19.332983+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"172'95","reported_seq":376,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-12-15T22:59:19.339304+0000","last_change":"2023-12-15T22:59:19.407340+0000","last_active":"2023-12-15T22:59:19.339304+0000","last_peered":"2023-12-15T22:59:19.339304+0000","last_clean":"2023-12-15T22:59:19.339304+0000","last_became_active":"2023-12-15T22:56:39.532730+0000","last_became_peered":"2023-12-15T22:56:39.532730+0000","last_unstale":"2023-12-15T22:59:19.339304+0000","last_undegraded":"2023-12-15T22:59:19.339304+0000","last_fullsized":"2023-12-15T22:59:19.339304+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5865709,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":47,"num_read_kb":38482,"num_write":44,"num_write_kb":13247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":259,"reported_epoch":173,"state":"active+clean","last_fresh":"2023-12-15T22:59:20.315512+0000","last_change":"2023-12-15T22:56:33.655736+0000","last_active":"2023-12-15T22:59:20.315512+0000","last_peered":"2023-12-15T22:59:20.315512+0000","last_clean":"2023-12-15T22:59:20.315512+0000","last_became_active":"2023-12-15T22:56:33.652917+0000","last_became_peered":"2023-12-15T22:56:33.652917+0000","last_unstale":"2023-12-15T22:59:20.315512+0000","last_undegraded":"2023-12-15T22:59:20.315512+0000","last_fullsized":"2023-12-15T22:59:20.315512+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"172'139","reported_seq":434,"reported_epoch":173,"state":"active+clean+snaptrim","last_fresh":"2023-12-15T22:59:20.316231+0000","last_change":"2023-12-15T22:59:19.427821+0000","last_active":"2023-12-15T22:59:20.316231+0000","last_peered":"2023-12-15T22:59:20.316231+0000","last_clean":"2023-12-15T22:59:20.316231+0000","last_became_active":"2023-12-15T22:56:39.557252+0000","last_became_peered":"2023-12-15T22:56:39.557252+0000","last_unstale":"2023-12-15T22:59:20.316231+0000","last_undegraded":"2023-12-15T22:59:20.316231+0000","last_fullsized":"2023-12-15T22:59:20.316231+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5815255,"num_objects":13,"num_object_clones":9,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":82,"num_read_kb":60078,"num_write":73,"num_write_kb":21141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":247,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-12-15T22:59:19.326094+0000","last_change":"2023-12-15T22:56:33.632348+0000","last_active":"2023-12-15T22:59:19.326094+0000","last_peered":"2023-12-15T22:59:19.326094+0000","last_clean":"2023-12-15T22:59:19.326094+0000","last_became_active":"2023-12-15T22:56:33.629022+0000","last_became_peered":"2023-12-15T22:56:33.629022+0000","last_unstale":"2023-12-15T22:59:19.326094+0000","last_undegraded":"2023-12-15T22:59:19.326094+0000","last_fullsized":"2023-12-15T22:59:19.326094+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"172'51","reported_seq":351,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-12-15T22:59:19.508151+0000","last_change":"2023-12-15T22:59:19.339003+0000","last_active":"2023-12-15T22:59:19.508151+0000","last_peered":"2023-12-15T22:59:19.508151+0000","last_clean":"2023-12-15T22:59:19.508151+0000","last_became_active":"2023-12-15T22:56:39.535145+0000","last_became_peered":"2023-12-15T22:56:39.535145+0000","last_unstale":"2023-12-15T22:59:19.508151+0000","last_undegraded":"2023-12-15T22:59:19.508151+0000","last_fullsized":"2023-12-15T22:59:19.508151+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3327819,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":29391,"num_write":22,"num_write_kb":6675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":245,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-12-15T22:59:19.422178+0000","last_change":"2023-12-15T22:56:33.627005+0000","last_active":"2023-12-15T22:59:19.422178+0000","last_peered":"2023-12-15T22:59:19.422178+0000","last_clean":"2023-12-15T22:59:19.422178+0000","last_became_active":"2023-12-15T22:56:33.622181+0000","last_became_peered":"2023-12-15T22:56:33.622181+0000","last_unstale":"2023-12-15T22:59:19.422178+0000","last_undegraded":"2023-12-15T22:59:19.422178+0000","last_fullsized":"2023-12-15T22:59:19.422178+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":92170248,"num_objects":227,"num_object_clones":164,"num_object_copies":681,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":32,"num_read":1286,"num_read_kb":945439,"num_write":1055,"num_write_kb":306371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2188,"ondisk_log_size":2188,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705834,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":142440,"kb_used_data":111456,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94229400,"statfs":{"total":96636764160,"available":96490905600,"internally_reserved":0,"allocated":114130944,"data_stored":132373887,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705833,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":121596,"kb_used_data":90612,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94250244,"statfs":{"total":96636764160,"available":96512249856,"internally_reserved":0,"allocated":92786688,"data_stored":107550539,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705833,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":139200,"kb_used_data":108216,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94232640,"statfs":{"total":96636764160,"available":96494223360,"internally_reserved":0,"allocated":110813184,"data_stored":126659602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705834,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":143624,"kb_used_data":112640,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94228216,"statfs":{"total":96636764160,"available":96489693184,"internally_reserved":0,"allocated":115343360,"data_stored":130212498,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-15T22:59:22.964 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2023-12-15T22:59:22.964 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T22:59:22.964 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-12-15T22:59:22.964 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-12-15T22:59:23.070 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:23.066+0000 7f5fad19e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:23.073 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:23.073+0000 7f5fad19e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:23.085 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:59:23.085 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:59:23.085 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:59:23.086 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:59:23.086 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:59:23.086 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:59:23.086 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:59:23.086 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:59:23.086 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:59:23.086 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:59:23.086 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:59:23.086 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:59:23.088 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:59:23.089 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:59:23.089 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-12-15T22:59:23.195 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:23.190+0000 7f9fe8e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:23.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:23.198+0000 7f9fe8e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:23.210 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:59:23.210 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:59:23.210 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:59:23.210 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:59:23.210 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:59:23.210 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:59:23.210 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:59:23.211 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:59:23.211 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:59:23.211 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:59:23.211 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:59:23.211 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:59:23.213 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:59:23.213 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:59:23.213 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-12-15T22:59:23.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:23.315+0000 7f05aeb04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:23.325 INFO:tasks.rados.rados.0.smithi071.stdout:675: done (1 left) 2023-12-15T22:59:23.325 INFO:tasks.rados.rados.0.smithi071.stdout:676: snap_create 2023-12-15T22:59:23.325 INFO:tasks.rados.rados.0.smithi071.stdout:673: expect (ObjNum 312 snap 81 seq_num 312) 2023-12-15T22:59:23.327 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:23.328+0000 7f05aeb04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:23.345 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:59:23.345 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:59:23.345 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:59:23.345 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:59:23.345 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:59:23.346 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:59:23.346 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:59:23.346 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:59:23.346 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:59:23.346 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:59:23.346 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:59:23.346 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:59:23.349 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:59:23.349 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:59:23.349 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-12-15T22:59:23.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:23.458+0000 7f2214111640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:23.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:23.466+0000 7f2214111640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:23.478 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:59:23.478 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:59:23.478 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:59:23.478 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:59:23.479 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:59:23.479 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:59:23.479 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:59:23.479 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:59:23.479 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:59:23.479 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:59:23.479 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:59:23.479 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:59:23.481 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:59:23.481 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:59:24.322 INFO:tasks.rados.rados.0.smithi071.stdout:673: done (1 left) 2023-12-15T22:59:24.322 INFO:tasks.rados.rados.0.smithi071.stdout:676: done (0 left) 2023-12-15T22:59:24.322 INFO:tasks.rados.rados.0.smithi071.stdout:677: read oid 9 snap -1 2023-12-15T22:59:24.322 INFO:tasks.rados.rados.0.smithi071.stdout:677: expect (ObjNum 321 snap 81 seq_num 321) 2023-12-15T22:59:24.323 INFO:tasks.rados.rados.0.smithi071.stdout:678: delete oid 16 current snap is 82 2023-12-15T22:59:24.422 INFO:tasks.rados.rados.0.smithi071.stdout:677: expect (ObjNum 321 snap 81 seq_num 321) 2023-12-15T22:59:24.423 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:59:24.538 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:24.537+0000 7fb2a239e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:24.541 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:24.541+0000 7fb2a239e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:24.697 INFO:tasks.rados.rados.0.smithi071.stdout:677: done (1 left) 2023-12-15T22:59:24.697 INFO:tasks.rados.rados.0.smithi071.stdout:678: done (0 left) 2023-12-15T22:59:24.697 INFO:tasks.rados.rados.0.smithi071.stdout:679: read oid 21 snap 80 2023-12-15T22:59:24.698 INFO:tasks.rados.rados.0.smithi071.stdout:679: expect deleted 2023-12-15T22:59:24.698 INFO:tasks.rados.rados.0.smithi071.stdout:680: read oid 14 snap 77 2023-12-15T22:59:24.698 INFO:tasks.rados.rados.0.smithi071.stdout:680: expect (ObjNum 232 snap 55 seq_num 232) 2023-12-15T22:59:24.698 INFO:tasks.rados.rados.0.smithi071.stdout:681: snap_remove snap 58 2023-12-15T22:59:25.330 INFO:tasks.rados.rados.0.smithi071.stdout:681: done (2 left) 2023-12-15T22:59:25.331 INFO:tasks.rados.rados.0.smithi071.stdout:682: snap_create 2023-12-15T22:59:25.331 INFO:tasks.rados.rados.0.smithi071.stdout:680: expect (ObjNum 232 snap 55 seq_num 232) 2023-12-15T22:59:26.331 INFO:tasks.rados.rados.0.smithi071.stdout:679: done (2 left) 2023-12-15T22:59:26.331 INFO:tasks.rados.rados.0.smithi071.stdout:680: done (1 left) 2023-12-15T22:59:26.331 INFO:tasks.rados.rados.0.smithi071.stdout:682: done (0 left) 2023-12-15T22:59:26.332 INFO:tasks.rados.rados.0.smithi071.stdout:683: read oid 46 snap -1 2023-12-15T22:59:26.332 INFO:tasks.rados.rados.0.smithi071.stdout:683: expect (ObjNum 277 snap 74 seq_num 277) 2023-12-15T22:59:26.332 INFO:tasks.rados.rados.0.smithi071.stdout:684: write (excl) oid 9 current snap is 83 2023-12-15T22:59:26.332 INFO:tasks.rados.rados.0.smithi071.stdout:684: seq_num 330 ranges {746310=635349,2077646=576443,3362764=183546} 2023-12-15T22:59:26.338 INFO:tasks.rados.rados.0.smithi071.stdout:684: writing smithi07183189-9 from 746310 to 1381659 tid 1 2023-12-15T22:59:26.346 INFO:tasks.rados.rados.0.smithi071.stdout:684: writing smithi07183189-9 from 2077646 to 2654089 tid 2 2023-12-15T22:59:26.351 INFO:tasks.rados.rados.0.smithi071.stdout:684: writing smithi07183189-9 from 3362764 to 3546310 tid 3 2023-12-15T22:59:26.351 INFO:tasks.rados.rados.0.smithi071.stdout:685: write oid 28 current snap is 83 2023-12-15T22:59:26.351 INFO:tasks.rados.rados.0.smithi071.stdout:685: seq_num 331 ranges {763117=710598,2136128=720101,3390187=172930} 2023-12-15T22:59:26.359 INFO:tasks.rados.rados.0.smithi071.stdout:685: writing smithi07183189-28 from 763117 to 1473715 tid 1 2023-12-15T22:59:26.367 INFO:tasks.rados.rados.0.smithi071.stdout:685: writing smithi07183189-28 from 2136128 to 2856229 tid 2 2023-12-15T22:59:26.371 INFO:tasks.rados.rados.0.smithi071.stdout:685: writing smithi07183189-28 from 3390187 to 3563117 tid 3 2023-12-15T22:59:26.371 INFO:tasks.rados.rados.0.smithi071.stdout:686: write (excl) oid 33 current snap is 83 2023-12-15T22:59:26.371 INFO:tasks.rados.rados.0.smithi071.stdout:686: seq_num 332 ranges {779924=785847,2078257=580112,3301257=278667} 2023-12-15T22:59:26.379 INFO:tasks.rados.rados.0.smithi071.stdout:686: writing smithi07183189-33 from 779924 to 1565771 tid 1 2023-12-15T22:59:26.386 INFO:tasks.rados.rados.0.smithi071.stdout:686: writing smithi07183189-33 from 2078257 to 2658369 tid 2 2023-12-15T22:59:26.391 INFO:tasks.rados.rados.0.smithi071.stdout:686: writing smithi07183189-33 from 3301257 to 3579924 tid 3 2023-12-15T22:59:26.391 INFO:tasks.rados.rados.0.smithi071.stdout:687: delete oid 24 current snap is 83 2023-12-15T22:59:26.476 INFO:tasks.rados.rados.0.smithi071.stdout:683: expect (ObjNum 277 snap 74 seq_num 277) 2023-12-15T22:59:26.591 INFO:tasks.rados.rados.0.smithi071.stdout:684: finishing write tid 1 to smithi07183189-9 2023-12-15T22:59:26.591 INFO:tasks.rados.rados.0.smithi071.stdout:686: finishing write tid 1 to smithi07183189-33 2023-12-15T22:59:26.591 INFO:tasks.rados.rados.0.smithi071.stdout:686: finishing write tid 2 to smithi07183189-33 2023-12-15T22:59:26.591 INFO:tasks.rados.rados.0.smithi071.stdout:684: finishing write tid 2 to smithi07183189-9 2023-12-15T22:59:26.604 INFO:tasks.rados.rados.0.smithi071.stdout:686: finishing write tid 3 to smithi07183189-33 2023-12-15T22:59:26.608 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:59:26.621 INFO:tasks.rados.rados.0.smithi071.stdout:683: done (4 left) 2023-12-15T22:59:26.621 INFO:tasks.rados.rados.0.smithi071.stdout:687: done (3 left) 2023-12-15T22:59:26.621 INFO:tasks.rados.rados.0.smithi071.stdout:688: write (excl) oid 44 current snap is 83 2023-12-15T22:59:26.621 INFO:tasks.rados.rados.0.smithi071.stdout:688: seq_num 334 ranges {413538=536345,1395221=583781,2556103=658617} 2023-12-15T22:59:26.628 INFO:tasks.rados.rados.0.smithi071.stdout:688: writing smithi07183189-44 from 413538 to 949883 tid 1 2023-12-15T22:59:26.634 INFO:tasks.rados.rados.0.smithi071.stdout:688: writing smithi07183189-44 from 1395221 to 1979002 tid 2 2023-12-15T22:59:26.642 INFO:tasks.rados.rados.0.smithi071.stdout:688: writing smithi07183189-44 from 2556103 to 3214720 tid 3 2023-12-15T22:59:26.642 INFO:tasks.rados.rados.0.smithi071.stdout:689: delete oid 23 current snap is 83 2023-12-15T22:59:26.642 INFO:tasks.rados.rados.0.smithi071.stdout:686: finishing write tid 4 to smithi07183189-33 2023-12-15T22:59:26.642 INFO:tasks.rados.rados.0.smithi071.stdout:686: finishing write tid 5 to smithi07183189-33 2023-12-15T22:59:26.642 INFO:tasks.rados.rados.0.smithi071.stdout:686: oid 33 updating version 0 to 505 2023-12-15T22:59:26.642 INFO:tasks.rados.rados.0.smithi071.stdout:686: oid 33 version 505 is already newer than 504 2023-12-15T22:59:26.642 INFO:tasks.rados.rados.0.smithi071.stdout:686: oid 33 updating version 505 to 507 2023-12-15T22:59:26.642 INFO:tasks.rados.rados.0.smithi071.stdout:686: oid 33 version 507 is already newer than 506 2023-12-15T22:59:26.642 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 33 v 507 (ObjNum 332 snap 83 seq_num 332) dirty exists 2023-12-15T22:59:26.642 INFO:tasks.rados.rados.0.smithi071.stdout:686: left oid 33 (ObjNum 332 snap 83 seq_num 332) 2023-12-15T22:59:26.655 INFO:tasks.rados.rados.0.smithi071.stdout:685: finishing write tid 1 to smithi07183189-28 2023-12-15T22:59:26.656 INFO:tasks.rados.rados.0.smithi071.stdout:684: finishing write tid 3 to smithi07183189-9 2023-12-15T22:59:26.695 INFO:tasks.rados.rados.0.smithi071.stdout:688: finishing write tid 1 to smithi07183189-44 2023-12-15T22:59:26.705 INFO:tasks.rados.rados.0.smithi071.stdout:685: finishing write tid 2 to smithi07183189-28 2023-12-15T22:59:26.705 INFO:tasks.rados.rados.0.smithi071.stdout:684: finishing write tid 4 to smithi07183189-9 2023-12-15T22:59:26.711 INFO:tasks.rados.rados.0.smithi071.stdout:684: finishing write tid 5 to smithi07183189-9 2023-12-15T22:59:26.711 INFO:tasks.rados.rados.0.smithi071.stdout:684: oid 9 updating version 0 to 220 2023-12-15T22:59:26.711 INFO:tasks.rados.rados.0.smithi071.stdout:684: oid 9 version 220 is already newer than 219 2023-12-15T22:59:26.711 INFO:tasks.rados.rados.0.smithi071.stdout:684: oid 9 updating version 220 to 222 2023-12-15T22:59:26.711 INFO:tasks.rados.rados.0.smithi071.stdout:684: oid 9 version 222 is already newer than 221 2023-12-15T22:59:26.712 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 222 (ObjNum 330 snap 83 seq_num 330) dirty exists 2023-12-15T22:59:26.712 INFO:tasks.rados.rados.0.smithi071.stdout:684: left oid 9 (ObjNum 330 snap 83 seq_num 330) 2023-12-15T22:59:26.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:26.722+0000 7f55b751e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:26.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:26.726+0000 7f55b751e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:26.728 INFO:tasks.rados.rados.0.smithi071.stdout:688: finishing write tid 2 to smithi07183189-44 2023-12-15T22:59:26.738 INFO:tasks.rados.rados.0.smithi071.stdout:684: done (4 left) 2023-12-15T22:59:26.738 INFO:tasks.rados.rados.0.smithi071.stdout:686: done (3 left) 2023-12-15T22:59:26.738 INFO:tasks.rados.rados.0.smithi071.stdout:689: done (2 left) 2023-12-15T22:59:26.738 INFO:tasks.rados.rados.0.smithi071.stdout:690: snap_remove snap 80 2023-12-15T22:59:27.261 INFO:tasks.rados.rados.0.smithi071.stdout:690: done (2 left) 2023-12-15T22:59:27.261 INFO:tasks.rados.rados.0.smithi071.stdout:691: write oid 30 current snap is 83 2023-12-15T22:59:27.261 INFO:tasks.rados.rados.0.smithi071.stdout:685: finishing write tid 3 to smithi07183189-28 2023-12-15T22:59:27.261 INFO:tasks.rados.rados.0.smithi071.stdout:691: seq_num 336 ranges {447152=403196,1512185=587450,2727302=681161} 2023-12-15T22:59:27.265 INFO:tasks.rados.rados.0.smithi071.stdout:691: writing smithi07183189-30 from 447152 to 850348 tid 1 2023-12-15T22:59:27.272 INFO:tasks.rados.rados.0.smithi071.stdout:691: writing smithi07183189-30 from 1512185 to 2099635 tid 2 2023-12-15T22:59:27.279 INFO:tasks.rados.rados.0.smithi071.stdout:691: writing smithi07183189-30 from 2727302 to 3408463 tid 3 2023-12-15T22:59:27.280 INFO:tasks.rados.rados.0.smithi071.stdout:692: snap_remove snap 77 2023-12-15T22:59:28.267 INFO:tasks.rados.rados.0.smithi071.stdout:692: done (3 left) 2023-12-15T22:59:28.268 INFO:tasks.rados.rados.0.smithi071.stdout:693: snap_create 2023-12-15T22:59:28.268 INFO:tasks.rados.rados.0.smithi071.stdout:688: finishing write tid 3 to smithi07183189-44 2023-12-15T22:59:28.268 INFO:tasks.rados.rados.0.smithi071.stdout:685: finishing write tid 4 to smithi07183189-28 2023-12-15T22:59:28.268 INFO:tasks.rados.rados.0.smithi071.stdout:685: finishing write tid 5 to smithi07183189-28 2023-12-15T22:59:28.268 INFO:tasks.rados.rados.0.smithi071.stdout:685: oid 28 updating version 0 to 122 2023-12-15T22:59:28.268 INFO:tasks.rados.rados.0.smithi071.stdout:685: oid 28 version 122 is already newer than 119 2023-12-15T22:59:28.268 INFO:tasks.rados.rados.0.smithi071.stdout:685: oid 28 version 122 is already newer than 120 2023-12-15T22:59:28.268 INFO:tasks.rados.rados.0.smithi071.stdout:685: oid 28 version 122 is already newer than 121 2023-12-15T22:59:28.268 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 122 (ObjNum 331 snap 83 seq_num 331) dirty exists 2023-12-15T22:59:28.268 INFO:tasks.rados.rados.0.smithi071.stdout:685: left oid 28 (ObjNum 331 snap 83 seq_num 331) 2023-12-15T22:59:28.268 INFO:tasks.rados.rados.0.smithi071.stdout:688: finishing write tid 4 to smithi07183189-44 2023-12-15T22:59:28.268 INFO:tasks.rados.rados.0.smithi071.stdout:688: finishing write tid 5 to smithi07183189-44 2023-12-15T22:59:28.268 INFO:tasks.rados.rados.0.smithi071.stdout:688: oid 44 updating version 0 to 108 2023-12-15T22:59:28.268 INFO:tasks.rados.rados.0.smithi071.stdout:688: oid 44 version 108 is already newer than 106 2023-12-15T22:59:28.269 INFO:tasks.rados.rados.0.smithi071.stdout:688: oid 44 version 108 is already newer than 105 2023-12-15T22:59:28.269 INFO:tasks.rados.rados.0.smithi071.stdout:688: oid 44 version 108 is already newer than 107 2023-12-15T22:59:28.269 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 108 (ObjNum 334 snap 83 seq_num 334) dirty exists 2023-12-15T22:59:28.269 INFO:tasks.rados.rados.0.smithi071.stdout:688: left oid 44 (ObjNum 334 snap 83 seq_num 334) 2023-12-15T22:59:28.269 INFO:tasks.rados.rados.0.smithi071.stdout:691: finishing write tid 1 to smithi07183189-30 2023-12-15T22:59:28.269 INFO:tasks.rados.rados.0.smithi071.stdout:691: finishing write tid 2 to smithi07183189-30 2023-12-15T22:59:28.269 INFO:tasks.rados.rados.0.smithi071.stdout:691: finishing write tid 3 to smithi07183189-30 2023-12-15T22:59:28.269 INFO:tasks.rados.rados.0.smithi071.stdout:691: finishing write tid 4 to smithi07183189-30 2023-12-15T22:59:28.269 INFO:tasks.rados.rados.0.smithi071.stdout:691: finishing write tid 5 to smithi07183189-30 2023-12-15T22:59:28.269 INFO:tasks.rados.rados.0.smithi071.stdout:691: oid 30 updating version 0 to 93 2023-12-15T22:59:28.269 INFO:tasks.rados.rados.0.smithi071.stdout:691: oid 30 updating version 93 to 96 2023-12-15T22:59:28.269 INFO:tasks.rados.rados.0.smithi071.stdout:691: oid 30 version 96 is already newer than 95 2023-12-15T22:59:28.269 INFO:tasks.rados.rados.0.smithi071.stdout:691: oid 30 version 96 is already newer than 95 2023-12-15T22:59:28.269 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 96 (ObjNum 336 snap 83 seq_num 336) dirty exists 2023-12-15T22:59:28.269 INFO:tasks.rados.rados.0.smithi071.stdout:691: left oid 30 (ObjNum 336 snap 83 seq_num 336) 2023-12-15T22:59:28.483 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T22:59:28.483 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T22:59:28.483 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-12-15T22:59:28.483 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-12-15T22:59:28.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:28.632+0000 7f7ac926d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:28.640 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:28.640+0000 7f7ac926d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:28.683 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:59:28.683 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:59:28.683 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:59:28.683 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:59:28.683 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:59:28.683 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:59:28.684 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:59:28.684 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:59:28.684 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:59:28.684 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:59:28.684 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:59:28.684 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:59:28.686 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:59:28.687 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:59:28.687 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-12-15T22:59:28.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:28.791+0000 7faca695e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:28.798 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:28.798+0000 7faca695e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:28.805 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:59:28.811 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:59:28.811 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:59:28.811 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:59:28.811 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:59:28.811 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:59:28.811 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:59:28.811 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:59:28.811 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:59:28.811 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:59:28.811 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:59:28.811 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:59:28.811 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:59:28.814 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:59:28.814 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:59:28.814 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-12-15T22:59:28.909 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:28.907+0000 7f981959e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:28.911 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:28.912+0000 7f981959e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:28.975 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:28.970+0000 7ff4d559e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:28.978 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:28.978+0000 7ff4d559e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:28.991 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:59:28.991 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:59:28.991 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:59:28.991 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:59:28.991 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:59:28.991 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:59:28.991 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:59:28.991 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:59:28.991 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:59:28.991 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:59:28.992 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:59:28.992 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:59:28.993 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:59:28.994 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:59:28.994 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-12-15T22:59:29.141 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:29.136+0000 7f832246d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:29.143 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:29.144+0000 7f832246d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:29.155 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:59:29.155 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:59:29.155 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:59:29.155 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:59:29.156 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:59:29.156 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:59:29.156 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:59:29.156 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:59:29.156 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:59:29.156 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:59:29.156 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:59:29.156 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:59:29.158 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:59:29.159 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:59:29.269 INFO:tasks.rados.rados.0.smithi071.stdout:685: done (3 left) 2023-12-15T22:59:29.270 INFO:tasks.rados.rados.0.smithi071.stdout:688: done (2 left) 2023-12-15T22:59:29.270 INFO:tasks.rados.rados.0.smithi071.stdout:691: done (1 left) 2023-12-15T22:59:29.270 INFO:tasks.rados.rados.0.smithi071.stdout:693: done (0 left) 2023-12-15T22:59:29.270 INFO:tasks.rados.rados.0.smithi071.stdout:694: read oid 19 snap -1 2023-12-15T22:59:29.270 INFO:tasks.rados.rados.0.smithi071.stdout:694: expect deleted 2023-12-15T22:59:29.270 INFO:tasks.rados.rados.0.smithi071.stdout:695: read oid 30 snap -1 2023-12-15T22:59:29.270 INFO:tasks.rados.rados.0.smithi071.stdout:695: expect (ObjNum 336 snap 83 seq_num 336) 2023-12-15T22:59:29.270 INFO:tasks.rados.rados.0.smithi071.stdout:696: delete oid 42 current snap is 84 2023-12-15T22:59:29.304 INFO:tasks.rados.rados.0.smithi071.stdout:696: done (2 left) 2023-12-15T22:59:29.304 INFO:tasks.rados.rados.0.smithi071.stdout:697: read oid 23 snap 81 2023-12-15T22:59:29.304 INFO:tasks.rados.rados.0.smithi071.stdout:697: expect (ObjNum 206 snap 44 seq_num 206) 2023-12-15T22:59:29.305 INFO:tasks.rados.rados.0.smithi071.stdout:698: read oid 16 snap -1 2023-12-15T22:59:29.305 INFO:tasks.rados.rados.0.smithi071.stdout:698: expect deleted 2023-12-15T22:59:29.305 INFO:tasks.rados.rados.0.smithi071.stdout:699: read oid 26 snap -1 2023-12-15T22:59:29.305 INFO:tasks.rados.rados.0.smithi071.stdout:699: expect (ObjNum 259 snap 70 seq_num 259) 2023-12-15T22:59:29.305 INFO:tasks.rados.rados.0.smithi071.stdout:700: read oid 47 snap -1 2023-12-15T22:59:29.305 INFO:tasks.rados.rados.0.smithi071.stdout:700: expect deleted 2023-12-15T22:59:29.305 INFO:tasks.rados.rados.0.smithi071.stdout:701: snap_remove snap 82 2023-12-15T22:59:30.279 INFO:tasks.rados.rados.0.smithi071.stdout:701: done (6 left) 2023-12-15T22:59:30.280 INFO:tasks.rados.rados.0.smithi071.stdout:702: read oid 32 snap 78 2023-12-15T22:59:30.280 INFO:tasks.rados.rados.0.smithi071.stdout:702: expect (ObjNum 288 snap 78 seq_num 288) 2023-12-15T22:59:30.280 INFO:tasks.rados.rados.0.smithi071.stdout:703: delete oid 46 current snap is 84 2023-12-15T22:59:30.280 INFO:tasks.rados.rados.0.smithi071.stdout:695: expect (ObjNum 336 snap 83 seq_num 336) 2023-12-15T22:59:30.515 INFO:tasks.rados.rados.0.smithi071.stdout:697: expect (ObjNum 206 snap 44 seq_num 206) 2023-12-15T22:59:30.653 INFO:tasks.rados.rados.0.smithi071.stdout:699: expect (ObjNum 259 snap 70 seq_num 259) 2023-12-15T22:59:30.787 INFO:tasks.rados.rados.0.smithi071.stdout:702: expect (ObjNum 288 snap 78 seq_num 288) 2023-12-15T22:59:30.939 INFO:tasks.rados.rados.0.smithi071.stdout:694: done (7 left) 2023-12-15T22:59:30.939 INFO:tasks.rados.rados.0.smithi071.stdout:695: done (6 left) 2023-12-15T22:59:30.939 INFO:tasks.rados.rados.0.smithi071.stdout:697: done (5 left) 2023-12-15T22:59:30.939 INFO:tasks.rados.rados.0.smithi071.stdout:698: done (4 left) 2023-12-15T22:59:30.940 INFO:tasks.rados.rados.0.smithi071.stdout:699: done (3 left) 2023-12-15T22:59:30.940 INFO:tasks.rados.rados.0.smithi071.stdout:700: done (2 left) 2023-12-15T22:59:30.940 INFO:tasks.rados.rados.0.smithi071.stdout:702: done (1 left) 2023-12-15T22:59:30.940 INFO:tasks.rados.rados.0.smithi071.stdout:703: done (0 left) 2023-12-15T22:59:30.940 INFO:tasks.rados.rados.0.smithi071.stdout:704: snap_remove snap 78 2023-12-15T22:59:30.993 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:59:31.096 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:31.095+0000 7f356db9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:31.099 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:31.099+0000 7f356db9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:31.284 INFO:tasks.rados.rados.0.smithi071.stdout:704: done (0 left) 2023-12-15T22:59:31.284 INFO:tasks.rados.rados.0.smithi071.stdout:705: write oid 13 current snap is 84 2023-12-15T22:59:31.284 INFO:tasks.rados.rados.0.smithi071.stdout:705: seq_num 339 ranges {497573=628943,1687631=734777,3093218=604355} 2023-12-15T22:59:31.290 INFO:tasks.rados.rados.0.smithi071.stdout:705: writing smithi07183189-13 from 497573 to 1126516 tid 1 2023-12-15T22:59:31.297 INFO:tasks.rados.rados.0.smithi071.stdout:705: writing smithi07183189-13 from 1687631 to 2422408 tid 2 2023-12-15T22:59:31.304 INFO:tasks.rados.rados.0.smithi071.stdout:705: writing smithi07183189-13 from 3093218 to 3697573 tid 3 2023-12-15T22:59:31.305 INFO:tasks.rados.rados.0.smithi071.stdout:706: read oid 49 snap 81 2023-12-15T22:59:31.305 INFO:tasks.rados.rados.0.smithi071.stdout:706: expect (ObjNum 246 snap 68 seq_num 246) 2023-12-15T22:59:31.305 INFO:tasks.rados.rados.0.smithi071.stdout:707: read oid 16 snap -1 2023-12-15T22:59:31.305 INFO:tasks.rados.rados.0.smithi071.stdout:707: expect deleted 2023-12-15T22:59:31.305 INFO:tasks.rados.rados.0.smithi071.stdout:708: read oid 37 snap 81 2023-12-15T22:59:31.305 INFO:tasks.rados.rados.0.smithi071.stdout:708: expect deleted 2023-12-15T22:59:31.305 INFO:tasks.rados.rados.0.smithi071.stdout:709: read oid 10 snap -1 2023-12-15T22:59:31.305 INFO:tasks.rados.rados.0.smithi071.stdout:709: expect (ObjNum 147 snap 20 seq_num 147) 2023-12-15T22:59:31.305 INFO:tasks.rados.rados.0.smithi071.stdout:710: delete oid 3 current snap is 84 2023-12-15T22:59:31.410 INFO:tasks.rados.rados.0.smithi071.stdout:709: expect (ObjNum 147 snap 20 seq_num 147) 2023-12-15T22:59:31.521 INFO:tasks.rados.rados.0.smithi071.stdout:706: expect (ObjNum 246 snap 68 seq_num 246) 2023-12-15T22:59:31.654 INFO:tasks.rados.rados.0.smithi071.stdout:705: finishing write tid 1 to smithi07183189-13 2023-12-15T22:59:31.655 INFO:tasks.rados.rados.0.smithi071.stdout:705: finishing write tid 2 to smithi07183189-13 2023-12-15T22:59:31.655 INFO:tasks.rados.rados.0.smithi071.stdout:705: finishing write tid 3 to smithi07183189-13 2023-12-15T22:59:31.655 INFO:tasks.rados.rados.0.smithi071.stdout:705: finishing write tid 4 to smithi07183189-13 2023-12-15T22:59:31.655 INFO:tasks.rados.rados.0.smithi071.stdout:706: done (5 left) 2023-12-15T22:59:31.655 INFO:tasks.rados.rados.0.smithi071.stdout:707: done (4 left) 2023-12-15T22:59:31.655 INFO:tasks.rados.rados.0.smithi071.stdout:708: done (3 left) 2023-12-15T22:59:31.655 INFO:tasks.rados.rados.0.smithi071.stdout:709: done (2 left) 2023-12-15T22:59:31.655 INFO:tasks.rados.rados.0.smithi071.stdout:710: done (1 left) 2023-12-15T22:59:31.655 INFO:tasks.rados.rados.0.smithi071.stdout:711: write oid 9 current snap is 84 2023-12-15T22:59:31.655 INFO:tasks.rados.rados.0.smithi071.stdout:705: finishing write tid 5 to smithi07183189-13 2023-12-15T22:59:31.655 INFO:tasks.rados.rados.0.smithi071.stdout:705: oid 13 updating version 0 to 239 2023-12-15T22:59:31.655 INFO:tasks.rados.rados.0.smithi071.stdout:705: oid 13 version 239 is already newer than 239 2023-12-15T22:59:31.655 INFO:tasks.rados.rados.0.smithi071.stdout:705: oid 13 version 239 is already newer than 237 2023-12-15T22:59:31.655 INFO:tasks.rados.rados.0.smithi071.stdout:705: oid 13 updating version 239 to 240 2023-12-15T22:59:31.655 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 240 (ObjNum 339 snap 84 seq_num 339) dirty exists 2023-12-15T22:59:31.656 INFO:tasks.rados.rados.0.smithi071.stdout:705: left oid 13 (ObjNum 339 snap 84 seq_num 339) 2023-12-15T22:59:31.656 INFO:tasks.rados.rados.0.smithi071.stdout:711: seq_num 341 ranges {531187=779441,2088242=738446,3431711=299476} 2023-12-15T22:59:31.661 INFO:tasks.rados.rados.0.smithi071.stdout:711: writing smithi07183189-9 from 531187 to 1310628 tid 1 2023-12-15T22:59:31.670 INFO:tasks.rados.rados.0.smithi071.stdout:711: writing smithi07183189-9 from 2088242 to 2826688 tid 2 2023-12-15T22:59:31.676 INFO:tasks.rados.rados.0.smithi071.stdout:711: writing smithi07183189-9 from 3431711 to 3731187 tid 3 2023-12-15T22:59:31.676 INFO:tasks.rados.rados.0.smithi071.stdout:705: done (1 left) 2023-12-15T22:59:31.676 INFO:tasks.rados.rados.0.smithi071.stdout:712: snap_remove snap 81 2023-12-15T22:59:32.292 INFO:tasks.rados.rados.0.smithi071.stdout:712: done (1 left) 2023-12-15T22:59:32.292 INFO:tasks.rados.rados.0.smithi071.stdout:713: write (excl) oid 1 current snap is 84 2023-12-15T22:59:32.292 INFO:tasks.rados.rados.0.smithi071.stdout:711: finishing write tid 1 to smithi07183189-9 2023-12-15T22:59:32.292 INFO:tasks.rados.rados.0.smithi071.stdout:713: seq_num 342 ranges {547994=454690,1746724=482104,2659134=465146,3605976=142018} 2023-12-15T22:59:32.297 INFO:tasks.rados.rados.0.smithi071.stdout:713: writing smithi07183189-1 from 547994 to 1002684 tid 1 2023-12-15T22:59:32.304 INFO:tasks.rados.rados.0.smithi071.stdout:713: writing smithi07183189-1 from 1746724 to 2228828 tid 2 2023-12-15T22:59:32.309 INFO:tasks.rados.rados.0.smithi071.stdout:713: writing smithi07183189-1 from 2659134 to 3124280 tid 3 2023-12-15T22:59:32.313 INFO:tasks.rados.rados.0.smithi071.stdout:713: writing smithi07183189-1 from 3605976 to 3747994 tid 4 2023-12-15T22:59:32.313 INFO:tasks.rados.rados.0.smithi071.stdout:714: write oid 34 current snap is 84 2023-12-15T22:59:32.313 INFO:tasks.rados.rados.0.smithi071.stdout:711: finishing write tid 2 to smithi07183189-9 2023-12-15T22:59:32.313 INFO:tasks.rados.rados.0.smithi071.stdout:714: seq_num 343 ranges {564801=646292,1921559=742115,3202910=561891} 2023-12-15T22:59:32.320 INFO:tasks.rados.rados.0.smithi071.stdout:714: writing smithi07183189-34 from 564801 to 1211093 tid 1 2023-12-15T22:59:32.329 INFO:tasks.rados.rados.0.smithi071.stdout:714: writing smithi07183189-34 from 1921559 to 2663674 tid 2 2023-12-15T22:59:32.337 INFO:tasks.rados.rados.0.smithi071.stdout:714: writing smithi07183189-34 from 3202910 to 3764801 tid 3 2023-12-15T22:59:32.337 INFO:tasks.rados.rados.0.smithi071.stdout:715: read oid 8 snap 83 2023-12-15T22:59:32.337 INFO:tasks.rados.rados.0.smithi071.stdout:715: expect deleted 2023-12-15T22:59:32.337 INFO:tasks.rados.rados.0.smithi071.stdout:716: read oid 24 snap -1 2023-12-15T22:59:32.337 INFO:tasks.rados.rados.0.smithi071.stdout:716: expect deleted 2023-12-15T22:59:32.337 INFO:tasks.rados.rados.0.smithi071.stdout:717: snap_create 2023-12-15T22:59:32.337 INFO:tasks.rados.rados.0.smithi071.stdout:711: finishing write tid 3 to smithi07183189-9 2023-12-15T22:59:32.337 INFO:tasks.rados.rados.0.smithi071.stdout:711: finishing write tid 4 to smithi07183189-9 2023-12-15T22:59:32.337 INFO:tasks.rados.rados.0.smithi071.stdout:711: finishing write tid 5 to smithi07183189-9 2023-12-15T22:59:32.337 INFO:tasks.rados.rados.0.smithi071.stdout:711: oid 9 updating version 0 to 238 2023-12-15T22:59:32.337 INFO:tasks.rados.rados.0.smithi071.stdout:711: oid 9 updating version 238 to 239 2023-12-15T22:59:32.337 INFO:tasks.rados.rados.0.smithi071.stdout:711: oid 9 updating version 239 to 240 2023-12-15T22:59:32.337 INFO:tasks.rados.rados.0.smithi071.stdout:711: oid 9 updating version 240 to 241 2023-12-15T22:59:32.338 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 241 (ObjNum 341 snap 84 seq_num 341) dirty exists 2023-12-15T22:59:32.338 INFO:tasks.rados.rados.0.smithi071.stdout:711: left oid 9 (ObjNum 341 snap 84 seq_num 341) 2023-12-15T22:59:32.385 INFO:tasks.rados.rados.0.smithi071.stdout:713: finishing write tid 1 to smithi07183189-1 2023-12-15T22:59:32.423 INFO:tasks.rados.rados.0.smithi071.stdout:713: finishing write tid 2 to smithi07183189-1 2023-12-15T22:59:32.430 INFO:tasks.rados.rados.0.smithi071.stdout:714: finishing write tid 1 to smithi07183189-34 2023-12-15T22:59:32.433 INFO:tasks.rados.rados.0.smithi071.stdout:713: finishing write tid 3 to smithi07183189-1 2023-12-15T22:59:32.448 INFO:tasks.rados.rados.0.smithi071.stdout:713: finishing write tid 4 to smithi07183189-1 2023-12-15T22:59:32.449 INFO:tasks.rados.rados.0.smithi071.stdout:714: finishing write tid 2 to smithi07183189-34 2023-12-15T22:59:32.457 INFO:tasks.rados.rados.0.smithi071.stdout:713: finishing write tid 5 to smithi07183189-1 2023-12-15T22:59:32.458 INFO:tasks.rados.rados.0.smithi071.stdout:713: finishing write tid 6 to smithi07183189-1 2023-12-15T22:59:32.458 INFO:tasks.rados.rados.0.smithi071.stdout:713: oid 1 updating version 0 to 90 2023-12-15T22:59:32.458 INFO:tasks.rados.rados.0.smithi071.stdout:713: oid 1 version 90 is already newer than 88 2023-12-15T22:59:32.458 INFO:tasks.rados.rados.0.smithi071.stdout:713: oid 1 version 90 is already newer than 89 2023-12-15T22:59:32.458 INFO:tasks.rados.rados.0.smithi071.stdout:713: oid 1 version 90 is already newer than 86 2023-12-15T22:59:32.458 INFO:tasks.rados.rados.0.smithi071.stdout:713: oid 1 version 90 is already newer than 88 2023-12-15T22:59:32.458 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 1 v 90 (ObjNum 342 snap 84 seq_num 342) dirty exists 2023-12-15T22:59:32.459 INFO:tasks.rados.rados.0.smithi071.stdout:713: left oid 1 (ObjNum 342 snap 84 seq_num 342) 2023-12-15T22:59:32.463 INFO:tasks.rados.rados.0.smithi071.stdout:714: finishing write tid 3 to smithi07183189-34 2023-12-15T22:59:32.471 INFO:tasks.rados.rados.0.smithi071.stdout:714: finishing write tid 4 to smithi07183189-34 2023-12-15T22:59:32.472 INFO:tasks.rados.rados.0.smithi071.stdout:714: finishing write tid 5 to smithi07183189-34 2023-12-15T22:59:32.472 INFO:tasks.rados.rados.0.smithi071.stdout:714: oid 34 updating version 0 to 67 2023-12-15T22:59:32.472 INFO:tasks.rados.rados.0.smithi071.stdout:714: oid 34 updating version 67 to 68 2023-12-15T22:59:32.472 INFO:tasks.rados.rados.0.smithi071.stdout:714: oid 34 updating version 68 to 69 2023-12-15T22:59:32.472 INFO:tasks.rados.rados.0.smithi071.stdout:714: oid 34 updating version 69 to 70 2023-12-15T22:59:32.472 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 70 (ObjNum 343 snap 84 seq_num 343) dirty exists 2023-12-15T22:59:32.473 INFO:tasks.rados.rados.0.smithi071.stdout:714: left oid 34 (ObjNum 343 snap 84 seq_num 343) 2023-12-15T22:59:33.167 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:59:33.275 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:33.274+0000 7f5128817640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:33.278 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:33.278+0000 7f5128817640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:33.294 INFO:tasks.rados.rados.0.smithi071.stdout:711: done (5 left) 2023-12-15T22:59:33.295 INFO:tasks.rados.rados.0.smithi071.stdout:713: done (4 left) 2023-12-15T22:59:33.295 INFO:tasks.rados.rados.0.smithi071.stdout:714: done (3 left) 2023-12-15T22:59:33.295 INFO:tasks.rados.rados.0.smithi071.stdout:715: done (2 left) 2023-12-15T22:59:33.295 INFO:tasks.rados.rados.0.smithi071.stdout:716: done (1 left) 2023-12-15T22:59:33.295 INFO:tasks.rados.rados.0.smithi071.stdout:717: done (0 left) 2023-12-15T22:59:33.295 INFO:tasks.rados.rados.0.smithi071.stdout:718: delete oid 2 current snap is 85 2023-12-15T22:59:33.355 INFO:tasks.rados.rados.0.smithi071.stdout:718: done (0 left) 2023-12-15T22:59:33.355 INFO:tasks.rados.rados.0.smithi071.stdout:719: read oid 14 snap -1 2023-12-15T22:59:33.355 INFO:tasks.rados.rados.0.smithi071.stdout:719: expect deleted 2023-12-15T22:59:33.355 INFO:tasks.rados.rados.0.smithi071.stdout:720: read oid 50 snap 84 2023-12-15T22:59:33.355 INFO:tasks.rados.rados.0.smithi071.stdout:720: expect (ObjNum 301 snap 80 seq_num 301) 2023-12-15T22:59:33.355 INFO:tasks.rados.rados.0.smithi071.stdout:721: read oid 43 snap 84 2023-12-15T22:59:33.355 INFO:tasks.rados.rados.0.smithi071.stdout:721: expect (ObjNum 223 snap 50 seq_num 223) 2023-12-15T22:59:33.356 INFO:tasks.rados.rados.0.smithi071.stdout:722: snap_remove snap 83 2023-12-15T22:59:34.159 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T22:59:34.160 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T22:59:34.160 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-12-15T22:59:34.160 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-12-15T22:59:34.274 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:34.270+0000 7f439b99e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:34.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:34.277+0000 7f439b99e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:34.289 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:59:34.289 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:59:34.289 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:59:34.289 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:59:34.289 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:59:34.289 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:59:34.289 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:59:34.289 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:59:34.289 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:59:34.290 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:59:34.290 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:59:34.290 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:59:34.292 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:59:34.292 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:59:34.292 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-12-15T22:59:34.299 INFO:tasks.rados.rados.0.smithi071.stdout:722: done (3 left) 2023-12-15T22:59:34.299 INFO:tasks.rados.rados.0.smithi071.stdout:723: snap_create 2023-12-15T22:59:34.299 INFO:tasks.rados.rados.0.smithi071.stdout:720: expect (ObjNum 301 snap 80 seq_num 301) 2023-12-15T22:59:34.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:34.395+0000 7f075bd5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:34.403 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:34.403+0000 7f075bd5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:34.415 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:59:34.415 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:59:34.415 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:59:34.415 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:59:34.415 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:59:34.415 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:59:34.415 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:59:34.415 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:59:34.415 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:59:34.415 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:59:34.415 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:59:34.415 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:59:34.418 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:59:34.418 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:59:34.418 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-12-15T22:59:34.585 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:34.579+0000 7f1ba375e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:34.589 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:34.589+0000 7f1ba375e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:34.592 INFO:tasks.rados.rados.0.smithi071.stdout:721: expect (ObjNum 223 snap 50 seq_num 223) 2023-12-15T22:59:34.625 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:59:34.625 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:59:34.625 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:59:34.625 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:59:34.625 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:59:34.625 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:59:34.625 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:59:34.625 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:59:34.625 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:59:34.625 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:59:34.625 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:59:34.625 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:59:34.627 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:59:34.628 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:59:34.628 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-12-15T22:59:34.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:34.731+0000 7fea4e955640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:34.738 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:34.738+0000 7fea4e955640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:34.751 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:59:34.751 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:59:34.751 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:59:34.751 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:59:34.751 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:59:34.751 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:59:34.751 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:59:34.751 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:59:34.752 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:59:34.752 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:59:34.752 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:59:34.752 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:59:34.753 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:59:34.754 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:59:35.302 INFO:tasks.rados.rados.0.smithi071.stdout:719: done (3 left) 2023-12-15T22:59:35.303 INFO:tasks.rados.rados.0.smithi071.stdout:720: done (2 left) 2023-12-15T22:59:35.303 INFO:tasks.rados.rados.0.smithi071.stdout:721: done (1 left) 2023-12-15T22:59:35.303 INFO:tasks.rados.rados.0.smithi071.stdout:723: done (0 left) 2023-12-15T22:59:35.303 INFO:tasks.rados.rados.0.smithi071.stdout:724: snap_remove snap 84 2023-12-15T22:59:35.424 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:59:35.536 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:35.534+0000 7f206346d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:35.539 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:35.539+0000 7f206346d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:36.303 INFO:tasks.rados.rados.0.smithi071.stdout:724: done (0 left) 2023-12-15T22:59:36.303 INFO:tasks.rados.rados.0.smithi071.stdout:725: read oid 3 snap -1 2023-12-15T22:59:36.304 INFO:tasks.rados.rados.0.smithi071.stdout:725: expect deleted 2023-12-15T22:59:36.304 INFO:tasks.rados.rados.0.smithi071.stdout:726: write oid 12 current snap is 86 2023-12-15T22:59:36.304 INFO:tasks.rados.rados.0.smithi071.stdout:726: seq_num 345 ranges {598415=796790,1922170=745784,3141403=415315} 2023-12-15T22:59:36.311 INFO:tasks.rados.rados.0.smithi071.stdout:726: writing smithi07183189-12 from 598415 to 1395205 tid 1 2023-12-15T22:59:36.318 INFO:tasks.rados.rados.0.smithi071.stdout:726: writing smithi07183189-12 from 1922170 to 2667954 tid 2 2023-12-15T22:59:36.323 INFO:tasks.rados.rados.0.smithi071.stdout:726: writing smithi07183189-12 from 3141403 to 3556718 tid 3 2023-12-15T22:59:36.323 INFO:tasks.rados.rados.0.smithi071.stdout:727: write oid 43 current snap is 86 2023-12-15T22:59:36.324 INFO:tasks.rados.rados.0.smithi071.stdout:727: seq_num 346 ranges {615222=472039,1580652=489442,2768826=626587} 2023-12-15T22:59:36.329 INFO:tasks.rados.rados.0.smithi071.stdout:727: writing smithi07183189-43 from 615222 to 1087261 tid 1 2023-12-15T22:59:36.336 INFO:tasks.rados.rados.0.smithi071.stdout:727: writing smithi07183189-43 from 1580652 to 2070094 tid 2 2023-12-15T22:59:36.343 INFO:tasks.rados.rados.0.smithi071.stdout:727: writing smithi07183189-43 from 2768826 to 3395413 tid 3 2023-12-15T22:59:36.344 INFO:tasks.rados.rados.0.smithi071.stdout:728: snap_create 2023-12-15T22:59:36.364 INFO:tasks.rados.rados.0.smithi071.stdout:726: finishing write tid 1 to smithi07183189-12 2023-12-15T22:59:36.405 INFO:tasks.rados.rados.0.smithi071.stdout:727: finishing write tid 1 to smithi07183189-43 2023-12-15T22:59:36.406 INFO:tasks.rados.rados.0.smithi071.stdout:726: finishing write tid 2 to smithi07183189-12 2023-12-15T22:59:36.406 INFO:tasks.rados.rados.0.smithi071.stdout:726: finishing write tid 3 to smithi07183189-12 2023-12-15T22:59:36.425 INFO:tasks.rados.rados.0.smithi071.stdout:727: finishing write tid 2 to smithi07183189-43 2023-12-15T22:59:36.426 INFO:tasks.rados.rados.0.smithi071.stdout:726: finishing write tid 4 to smithi07183189-12 2023-12-15T22:59:36.427 INFO:tasks.rados.rados.0.smithi071.stdout:726: finishing write tid 5 to smithi07183189-12 2023-12-15T22:59:36.427 INFO:tasks.rados.rados.0.smithi071.stdout:726: oid 12 updating version 0 to 133 2023-12-15T22:59:36.427 INFO:tasks.rados.rados.0.smithi071.stdout:726: oid 12 updating version 133 to 135 2023-12-15T22:59:36.427 INFO:tasks.rados.rados.0.smithi071.stdout:726: oid 12 updating version 135 to 136 2023-12-15T22:59:36.427 INFO:tasks.rados.rados.0.smithi071.stdout:726: oid 12 version 136 is already newer than 135 2023-12-15T22:59:36.427 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 12 v 136 (ObjNum 345 snap 86 seq_num 345) dirty exists 2023-12-15T22:59:36.427 INFO:tasks.rados.rados.0.smithi071.stdout:726: left oid 12 (ObjNum 345 snap 86 seq_num 345) 2023-12-15T22:59:36.449 INFO:tasks.rados.rados.0.smithi071.stdout:727: finishing write tid 3 to smithi07183189-43 2023-12-15T22:59:36.449 INFO:tasks.rados.rados.0.smithi071.stdout:727: finishing write tid 4 to smithi07183189-43 2023-12-15T22:59:36.450 INFO:tasks.rados.rados.0.smithi071.stdout:727: finishing write tid 5 to smithi07183189-43 2023-12-15T22:59:36.450 INFO:tasks.rados.rados.0.smithi071.stdout:727: oid 43 updating version 0 to 106 2023-12-15T22:59:36.450 INFO:tasks.rados.rados.0.smithi071.stdout:727: oid 43 updating version 106 to 109 2023-12-15T22:59:36.451 INFO:tasks.rados.rados.0.smithi071.stdout:727: oid 43 version 109 is already newer than 107 2023-12-15T22:59:36.451 INFO:tasks.rados.rados.0.smithi071.stdout:727: oid 43 version 109 is already newer than 109 2023-12-15T22:59:36.451 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 109 (ObjNum 346 snap 86 seq_num 346) dirty exists 2023-12-15T22:59:36.451 INFO:tasks.rados.rados.0.smithi071.stdout:727: left oid 43 (ObjNum 346 snap 86 seq_num 346) 2023-12-15T22:59:37.264 INFO:tasks.rados.rados.0.smithi071.stdout:725: done (3 left) 2023-12-15T22:59:37.264 INFO:tasks.rados.rados.0.smithi071.stdout:726: done (2 left) 2023-12-15T22:59:37.264 INFO:tasks.rados.rados.0.smithi071.stdout:727: done (1 left) 2023-12-15T22:59:37.265 INFO:tasks.rados.rados.0.smithi071.stdout:728: done (0 left) 2023-12-15T22:59:37.265 INFO:tasks.rados.rados.0.smithi071.stdout:729: write (excl) oid 19 current snap is 87 2023-12-15T22:59:37.265 INFO:tasks.rados.rados.0.smithi071.stdout:729: seq_num 347 ranges {632029=547288,1639134=749453,2796249=437859,3653524=178505} 2023-12-15T22:59:37.270 INFO:tasks.rados.rados.0.smithi071.stdout:729: writing smithi07183189-19 from 632029 to 1179317 tid 1 2023-12-15T22:59:37.277 INFO:tasks.rados.rados.0.smithi071.stdout:729: writing smithi07183189-19 from 1639134 to 2388587 tid 2 2023-12-15T22:59:37.282 INFO:tasks.rados.rados.0.smithi071.stdout:729: writing smithi07183189-19 from 2796249 to 3234108 tid 3 2023-12-15T22:59:37.286 INFO:tasks.rados.rados.0.smithi071.stdout:729: writing smithi07183189-19 from 3653524 to 3832029 tid 4 2023-12-15T22:59:37.286 INFO:tasks.rados.rados.0.smithi071.stdout:730: write oid 9 current snap is 87 2023-12-15T22:59:37.286 INFO:tasks.rados.rados.0.smithi071.stdout:730: seq_num 348 ranges {648836=622537,1981263=493111,3107319=741517} 2023-12-15T22:59:37.293 INFO:tasks.rados.rados.0.smithi071.stdout:730: writing smithi07183189-9 from 648836 to 1271373 tid 1 2023-12-15T22:59:37.299 INFO:tasks.rados.rados.0.smithi071.stdout:730: writing smithi07183189-9 from 1981263 to 2474374 tid 2 2023-12-15T22:59:37.307 INFO:tasks.rados.rados.0.smithi071.stdout:730: writing smithi07183189-9 from 3107319 to 3848836 tid 3 2023-12-15T22:59:37.307 INFO:tasks.rados.rados.0.smithi071.stdout:731: delete oid 5 current snap is 87 2023-12-15T22:59:37.313 INFO:tasks.rados.rados.0.smithi071.stdout:729: finishing write tid 1 to smithi07183189-19 2023-12-15T22:59:37.335 INFO:tasks.rados.rados.0.smithi071.stdout:731: done (2 left) 2023-12-15T22:59:37.335 INFO:tasks.rados.rados.0.smithi071.stdout:732: snap_remove snap 85 2023-12-15T22:59:37.624 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:59:37.730 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:37.728+0000 7f6aebd1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:37.733 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:37.733+0000 7f6aebd1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:38.313 INFO:tasks.rados.rados.0.smithi071.stdout:732: done (2 left) 2023-12-15T22:59:38.313 INFO:tasks.rados.rados.0.smithi071.stdout:733: read oid 6 snap -1 2023-12-15T22:59:38.313 INFO:tasks.rados.rados.0.smithi071.stdout:733: expect (ObjNum 300 snap 80 seq_num 300) 2023-12-15T22:59:38.313 INFO:tasks.rados.rados.0.smithi071.stdout:734: snap_remove snap 86 2023-12-15T22:59:39.318 INFO:tasks.rados.rados.0.smithi071.stdout:734: done (3 left) 2023-12-15T22:59:39.319 INFO:tasks.rados.rados.0.smithi071.stdout:735: delete oid 2 current snap is 87 2023-12-15T22:59:39.319 INFO:tasks.rados.rados.0.smithi071.stdout:729: finishing write tid 2 to smithi07183189-19 2023-12-15T22:59:39.319 INFO:tasks.rados.rados.0.smithi071.stdout:729: finishing write tid 3 to smithi07183189-19 2023-12-15T22:59:39.319 INFO:tasks.rados.rados.0.smithi071.stdout:729: finishing write tid 4 to smithi07183189-19 2023-12-15T22:59:39.319 INFO:tasks.rados.rados.0.smithi071.stdout:729: finishing write tid 5 to smithi07183189-19 2023-12-15T22:59:39.319 INFO:tasks.rados.rados.0.smithi071.stdout:729: finishing write tid 6 to smithi07183189-19 2023-12-15T22:59:39.319 INFO:tasks.rados.rados.0.smithi071.stdout:729: oid 19 updating version 0 to 250 2023-12-15T22:59:39.319 INFO:tasks.rados.rados.0.smithi071.stdout:729: oid 19 version 250 is already newer than 247 2023-12-15T22:59:39.319 INFO:tasks.rados.rados.0.smithi071.stdout:729: oid 19 version 250 is already newer than 249 2023-12-15T22:59:39.319 INFO:tasks.rados.rados.0.smithi071.stdout:729: oid 19 version 250 is already newer than 248 2023-12-15T22:59:39.319 INFO:tasks.rados.rados.0.smithi071.stdout:729: oid 19 version 250 is already newer than 246 2023-12-15T22:59:39.319 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 250 (ObjNum 347 snap 87 seq_num 347) dirty exists 2023-12-15T22:59:39.319 INFO:tasks.rados.rados.0.smithi071.stdout:729: left oid 19 (ObjNum 347 snap 87 seq_num 347) 2023-12-15T22:59:39.319 INFO:tasks.rados.rados.0.smithi071.stdout:730: finishing write tid 1 to smithi07183189-9 2023-12-15T22:59:39.320 INFO:tasks.rados.rados.0.smithi071.stdout:730: finishing write tid 2 to smithi07183189-9 2023-12-15T22:59:39.320 INFO:tasks.rados.rados.0.smithi071.stdout:730: finishing write tid 3 to smithi07183189-9 2023-12-15T22:59:39.320 INFO:tasks.rados.rados.0.smithi071.stdout:730: finishing write tid 4 to smithi07183189-9 2023-12-15T22:59:39.320 INFO:tasks.rados.rados.0.smithi071.stdout:730: finishing write tid 5 to smithi07183189-9 2023-12-15T22:59:39.320 INFO:tasks.rados.rados.0.smithi071.stdout:730: oid 9 updating version 0 to 255 2023-12-15T22:59:39.320 INFO:tasks.rados.rados.0.smithi071.stdout:730: oid 9 version 255 is already newer than 255 2023-12-15T22:59:39.320 INFO:tasks.rados.rados.0.smithi071.stdout:730: oid 9 version 255 is already newer than 255 2023-12-15T22:59:39.320 INFO:tasks.rados.rados.0.smithi071.stdout:730: oid 9 version 255 is already newer than 255 2023-12-15T22:59:39.320 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 255 (ObjNum 348 snap 87 seq_num 348) dirty exists 2023-12-15T22:59:39.320 INFO:tasks.rados.rados.0.smithi071.stdout:730: left oid 9 (ObjNum 348 snap 87 seq_num 348) 2023-12-15T22:59:39.320 INFO:tasks.rados.rados.0.smithi071.stdout:733: expect (ObjNum 300 snap 80 seq_num 300) 2023-12-15T22:59:39.571 INFO:tasks.rados.rados.0.smithi071.stdout:729: done (3 left) 2023-12-15T22:59:39.571 INFO:tasks.rados.rados.0.smithi071.stdout:730: done (2 left) 2023-12-15T22:59:39.571 INFO:tasks.rados.rados.0.smithi071.stdout:733: done (1 left) 2023-12-15T22:59:39.571 INFO:tasks.rados.rados.0.smithi071.stdout:735: done (0 left) 2023-12-15T22:59:39.571 INFO:tasks.rados.rados.0.smithi071.stdout:736: snap_create 2023-12-15T22:59:39.755 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T22:59:39.755 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2023-12-15T22:59:39.800 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:59:39.865 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:39.860+0000 7f325919e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:39.868 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:39.868+0000 7f325919e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:39.908 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:39.907+0000 7f41bfd11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:39.911 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:39.911+0000 7f41bfd11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:40.316 INFO:teuthology.orchestra.run.smithi071.stderr:reweighted osd.0 to 1 (10000) 2023-12-15T22:59:40.319 INFO:tasks.rados.rados.0.smithi071.stdout:736: done (0 left) 2023-12-15T22:59:40.319 INFO:tasks.rados.rados.0.smithi071.stdout:737: write oid 12 current snap is 88 2023-12-15T22:59:40.319 INFO:tasks.rados.rados.0.smithi071.stdout:737: seq_num 351 ranges {699257=564637,1873062=640438,3305941=593316} 2023-12-15T22:59:40.326 INFO:tasks.rados.rados.0.smithi071.stdout:737: writing smithi07183189-12 from 699257 to 1263894 tid 1 2023-12-15T22:59:40.329 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2023-12-15T22:59:40.333 INFO:tasks.rados.rados.0.smithi071.stdout:737: writing smithi07183189-12 from 1873062 to 2513500 tid 2 2023-12-15T22:59:40.341 INFO:tasks.rados.rados.0.smithi071.stdout:737: writing smithi07183189-12 from 3305941 to 3899257 tid 3 2023-12-15T22:59:40.341 INFO:tasks.rados.rados.0.smithi071.stdout:738: snap_remove snap 87 2023-12-15T22:59:40.445 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:40.440+0000 7f6dabe6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:40.448 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:40.448+0000 7f6dabe6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:41.324 INFO:teuthology.orchestra.run.smithi071.stderr:reweighted osd.1 to 1 (10000) 2023-12-15T22:59:41.331 INFO:tasks.rados.rados.0.smithi071.stdout:738: done (1 left) 2023-12-15T22:59:41.331 INFO:tasks.rados.rados.0.smithi071.stdout:739: delete oid 31 current snap is 88 2023-12-15T22:59:41.331 INFO:tasks.rados.rados.0.smithi071.stdout:737: finishing write tid 1 to smithi07183189-12 2023-12-15T22:59:41.332 INFO:tasks.rados.rados.0.smithi071.stdout:737: finishing write tid 2 to smithi07183189-12 2023-12-15T22:59:41.332 INFO:tasks.rados.rados.0.smithi071.stdout:737: finishing write tid 3 to smithi07183189-12 2023-12-15T22:59:41.332 INFO:tasks.rados.rados.0.smithi071.stdout:737: finishing write tid 4 to smithi07183189-12 2023-12-15T22:59:41.332 INFO:tasks.rados.rados.0.smithi071.stdout:737: finishing write tid 5 to smithi07183189-12 2023-12-15T22:59:41.332 INFO:tasks.rados.rados.0.smithi071.stdout:737: oid 12 updating version 0 to 138 2023-12-15T22:59:41.332 INFO:tasks.rados.rados.0.smithi071.stdout:737: oid 12 updating version 138 to 141 2023-12-15T22:59:41.332 INFO:tasks.rados.rados.0.smithi071.stdout:737: oid 12 version 141 is already newer than 139 2023-12-15T22:59:41.332 INFO:tasks.rados.rados.0.smithi071.stdout:737: oid 12 version 141 is already newer than 140 2023-12-15T22:59:41.332 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 12 v 141 (ObjNum 351 snap 88 seq_num 351) dirty exists 2023-12-15T22:59:41.332 INFO:tasks.rados.rados.0.smithi071.stdout:737: left oid 12 (ObjNum 351 snap 88 seq_num 351) 2023-12-15T22:59:41.338 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2023-12-15T22:59:41.360 INFO:tasks.rados.rados.0.smithi071.stdout:737: done (1 left) 2023-12-15T22:59:41.360 INFO:tasks.rados.rados.0.smithi071.stdout:739: done (0 left) 2023-12-15T22:59:41.360 INFO:tasks.rados.rados.0.smithi071.stdout:740: write (excl) oid 50 current snap is 88 2023-12-15T22:59:41.360 INFO:tasks.rados.rados.0.smithi071.stdout:740: seq_num 353 ranges {732871=715135,1873673=644107,3244434=688437} 2023-12-15T22:59:41.368 INFO:tasks.rados.rados.0.smithi071.stdout:740: writing smithi07183189-50 from 732871 to 1448006 tid 1 2023-12-15T22:59:41.375 INFO:tasks.rados.rados.0.smithi071.stdout:740: writing smithi07183189-50 from 1873673 to 2517780 tid 2 2023-12-15T22:59:41.383 INFO:tasks.rados.rados.0.smithi071.stdout:740: writing smithi07183189-50 from 3244434 to 3932871 tid 3 2023-12-15T22:59:41.383 INFO:tasks.rados.rados.0.smithi071.stdout:741: snap_create 2023-12-15T22:59:41.430 INFO:tasks.rados.rados.0.smithi071.stdout:740: finishing write tid 1 to smithi07183189-50 2023-12-15T22:59:41.452 INFO:tasks.rados.rados.0.smithi071.stdout:740: finishing write tid 2 to smithi07183189-50 2023-12-15T22:59:41.474 INFO:tasks.rados.rados.0.smithi071.stdout:740: finishing write tid 3 to smithi07183189-50 2023-12-15T22:59:41.478 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:41.473+0000 7f52ef79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:41.482 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:41.482+0000 7f52ef79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:41.483 INFO:tasks.rados.rados.0.smithi071.stdout:740: finishing write tid 4 to smithi07183189-50 2023-12-15T22:59:41.485 INFO:tasks.rados.rados.0.smithi071.stdout:740: finishing write tid 5 to smithi07183189-50 2023-12-15T22:59:41.485 INFO:tasks.rados.rados.0.smithi071.stdout:740: oid 50 updating version 0 to 539 2023-12-15T22:59:41.485 INFO:tasks.rados.rados.0.smithi071.stdout:740: oid 50 version 539 is already newer than 536 2023-12-15T22:59:41.485 INFO:tasks.rados.rados.0.smithi071.stdout:740: oid 50 version 539 is already newer than 538 2023-12-15T22:59:41.485 INFO:tasks.rados.rados.0.smithi071.stdout:740: oid 50 version 539 is already newer than 537 2023-12-15T22:59:41.485 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 539 (ObjNum 353 snap 88 seq_num 353) dirty exists 2023-12-15T22:59:41.485 INFO:tasks.rados.rados.0.smithi071.stdout:740: left oid 50 (ObjNum 353 snap 88 seq_num 353) 2023-12-15T22:59:41.982 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:59:42.086 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:42.085+0000 7ff1de66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:42.089 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:42.089+0000 7ff1de66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:42.328 INFO:teuthology.orchestra.run.smithi071.stderr:reweighted osd.2 to 1 (10000) 2023-12-15T22:59:42.331 INFO:tasks.rados.rados.0.smithi071.stdout:740: done (1 left) 2023-12-15T22:59:42.331 INFO:tasks.rados.rados.0.smithi071.stdout:741: done (0 left) 2023-12-15T22:59:42.331 INFO:tasks.rados.rados.0.smithi071.stdout:742: read oid 17 snap -1 2023-12-15T22:59:42.331 INFO:tasks.rados.rados.0.smithi071.stdout:742: expect (ObjNum 220 snap 49 seq_num 220) 2023-12-15T22:59:42.331 INFO:tasks.rados.rados.0.smithi071.stdout:743: snap_create 2023-12-15T22:59:42.344 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2023-12-15T22:59:42.375 INFO:tasks.rados.rados.0.smithi071.stdout:742: expect (ObjNum 220 snap 49 seq_num 220) 2023-12-15T22:59:42.453 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:42.448+0000 7fb00e82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:42.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:42.456+0000 7fb00e82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:43.332 INFO:teuthology.orchestra.run.smithi071.stderr:reweighted osd.3 to 1 (10000) 2023-12-15T22:59:43.340 INFO:tasks.rados.rados.0.smithi071.stdout:742: done (1 left) 2023-12-15T22:59:43.340 INFO:tasks.rados.rados.0.smithi071.stdout:743: done (0 left) 2023-12-15T22:59:43.340 INFO:tasks.rados.rados.0.smithi071.stdout:744: read oid 29 snap -1 2023-12-15T22:59:43.340 INFO:tasks.rados.rados.0.smithi071.stdout:744: expect (ObjNum 293 snap 80 seq_num 293) 2023-12-15T22:59:43.340 INFO:tasks.rados.rados.0.smithi071.stdout:745: write oid 35 current snap is 90 2023-12-15T22:59:43.340 INFO:tasks.rados.rados.0.smithi071.stdout:745: seq_num 354 ranges {749678=790384,2332155=504118,3271857=549469} 2023-12-15T22:59:43.349 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2023-12-15T22:59:43.349 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T22:59:43.350 INFO:tasks.rados.rados.0.smithi071.stdout:745: writing smithi07183189-35 from 749678 to 1540062 tid 1 2023-12-15T22:59:43.358 INFO:tasks.rados.rados.0.smithi071.stdout:745: writing smithi07183189-35 from 2332155 to 2836273 tid 2 2023-12-15T22:59:43.365 INFO:tasks.rados.rados.0.smithi071.stdout:745: writing smithi07183189-35 from 3271857 to 3821326 tid 3 2023-12-15T22:59:43.365 INFO:tasks.rados.rados.0.smithi071.stdout:746: read oid 46 snap -1 2023-12-15T22:59:43.365 INFO:tasks.rados.rados.0.smithi071.stdout:746: expect deleted 2023-12-15T22:59:43.365 INFO:tasks.rados.rados.0.smithi071.stdout:747: read oid 9 snap -1 2023-12-15T22:59:43.365 INFO:tasks.rados.rados.0.smithi071.stdout:747: expect (ObjNum 348 snap 87 seq_num 348) 2023-12-15T22:59:43.365 INFO:tasks.rados.rados.0.smithi071.stdout:748: delete oid 30 current snap is 90 2023-12-15T22:59:43.405 INFO:tasks.rados.rados.0.smithi071.stdout:744: expect (ObjNum 293 snap 80 seq_num 293) 2023-12-15T22:59:43.472 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:43.469+0000 7f0bebf1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:43.476 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:43.476+0000 7f0bebf1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:43.560 INFO:tasks.rados.rados.0.smithi071.stdout:744: done (4 left) 2023-12-15T22:59:43.560 INFO:tasks.rados.rados.0.smithi071.stdout:746: done (3 left) 2023-12-15T22:59:43.561 INFO:tasks.rados.rados.0.smithi071.stdout:748: done (2 left) 2023-12-15T22:59:43.561 INFO:tasks.rados.rados.0.smithi071.stdout:749: write oid 19 current snap is 90 2023-12-15T22:59:43.561 INFO:tasks.rados.rados.0.smithi071.stdout:745: finishing write tid 1 to smithi07183189-35 2023-12-15T22:59:43.561 INFO:tasks.rados.rados.0.smithi071.stdout:745: finishing write tid 2 to smithi07183189-35 2023-12-15T22:59:43.561 INFO:tasks.rados.rados.0.smithi071.stdout:745: finishing write tid 3 to smithi07183189-35 2023-12-15T22:59:43.561 INFO:tasks.rados.rados.0.smithi071.stdout:745: finishing write tid 4 to smithi07183189-35 2023-12-15T22:59:43.561 INFO:tasks.rados.rados.0.smithi071.stdout:745: finishing write tid 5 to smithi07183189-35 2023-12-15T22:59:43.561 INFO:tasks.rados.rados.0.smithi071.stdout:745: oid 35 updating version 0 to 262 2023-12-15T22:59:43.561 INFO:tasks.rados.rados.0.smithi071.stdout:745: oid 35 version 262 is already newer than 260 2023-12-15T22:59:43.561 INFO:tasks.rados.rados.0.smithi071.stdout:745: oid 35 version 262 is already newer than 261 2023-12-15T22:59:43.561 INFO:tasks.rados.rados.0.smithi071.stdout:745: oid 35 version 262 is already newer than 259 2023-12-15T22:59:43.561 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 262 (ObjNum 354 snap 90 seq_num 354) dirty exists 2023-12-15T22:59:43.561 INFO:tasks.rados.rados.0.smithi071.stdout:745: left oid 35 (ObjNum 354 snap 90 seq_num 354) 2023-12-15T22:59:43.562 INFO:tasks.rados.rados.0.smithi071.stdout:747: expect (ObjNum 348 snap 87 seq_num 348) 2023-12-15T22:59:43.662 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:59:43.662 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-12-15T22:59:43.674 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":286,"stamp":"2023-12-15T22:59:43.335781+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":91130403,"num_objects":252,"num_object_clones":183,"num_object_copies":756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":39,"num_read":1521,"num_read_kb":1069219,"num_write":1279,"num_write_kb":336594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2802,"ondisk_log_size":2802,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":299412,"kb_used_data":171316,"kb_used_omap":0,"kb_used_meta":128064,"kb_avail":377187948,"statfs":{"total":386547056640,"available":386240458752,"internally_reserved":0,"allocated":175427584,"data_stored":178017628,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":131137536},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5042166,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":12,"num_read_kb":8918,"num_write":26,"num_write_kb":8908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.018855"},"pg_stats":[{"pgid":"3.f","version":"185'90","reported_seq":453,"reported_epoch":193,"state":"active+clean","last_fresh":"2023-12-15T22:59:40.564752+0000","last_change":"2023-12-15T22:59:39.357157+0000","last_active":"2023-12-15T22:59:40.564752+0000","last_peered":"2023-12-15T22:59:40.564752+0000","last_clean":"2023-12-15T22:59:40.564752+0000","last_became_active":"2023-12-15T22:56:39.503797+0000","last_became_peered":"2023-12-15T22:56:39.503797+0000","last_unstale":"2023-12-15T22:59:40.564752+0000","last_undegraded":"2023-12-15T22:59:40.564752+0000","last_fullsized":"2023-12-15T22:59:40.564752+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6991778,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":70,"num_read_kb":53991,"num_write":36,"num_write_kb":10667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":360,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-12-15T22:59:37.663795+0000","last_change":"2023-12-15T22:59:36.656108+0000","last_active":"2023-12-15T22:59:37.663795+0000","last_peered":"2023-12-15T22:59:37.663795+0000","last_clean":"2023-12-15T22:59:37.663795+0000","last_became_active":"2023-12-15T22:56:39.552856+0000","last_became_peered":"2023-12-15T22:56:39.552856+0000","last_unstale":"2023-12-15T22:59:37.663795+0000","last_undegraded":"2023-12-15T22:59:37.663795+0000","last_fullsized":"2023-12-15T22:59:37.663795+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"185'263","reported_seq":571,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-12-15T22:59:38.325831+0000","last_change":"2023-12-15T22:59:38.330714+0000","last_active":"2023-12-15T22:59:38.325831+0000","last_peered":"2023-12-15T22:59:38.325831+0000","last_clean":"2023-12-15T22:59:38.325831+0000","last_became_active":"2023-12-15T22:56:39.495490+0000","last_became_peered":"2023-12-15T22:56:39.495490+0000","last_unstale":"2023-12-15T22:59:38.325831+0000","last_undegraded":"2023-12-15T22:59:38.325831+0000","last_fullsized":"2023-12-15T22:59:38.325831+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":263,"log_dups_size":0,"ondisk_log_size":263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10101040,"num_objects":27,"num_object_clones":18,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":199,"num_read_kb":180268,"num_write":132,"num_write_kb":38041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"192'257","reported_seq":558,"reported_epoch":193,"state":"active+clean+snaptrim","last_fresh":"2023-12-15T22:59:40.567511+0000","last_change":"2023-12-15T22:59:39.349202+0000","last_active":"2023-12-15T22:59:40.567511+0000","last_peered":"2023-12-15T22:59:40.567511+0000","last_clean":"2023-12-15T22:59:40.567511+0000","last_became_active":"2023-12-15T22:56:39.579234+0000","last_became_peered":"2023-12-15T22:56:39.579234+0000","last_unstale":"2023-12-15T22:59:40.567511+0000","last_undegraded":"2023-12-15T22:59:40.567511+0000","last_fullsized":"2023-12-15T22:59:40.567511+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10252369,"num_objects":24,"num_object_clones":20,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":136,"num_read_kb":98667,"num_write":125,"num_write_kb":36167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"193'141","reported_seq":457,"reported_epoch":193,"state":"active+clean","last_fresh":"2023-12-15T22:59:40.435038+0000","last_change":"2023-12-15T22:59:40.361645+0000","last_active":"2023-12-15T22:59:40.435038+0000","last_peered":"2023-12-15T22:59:40.435038+0000","last_clean":"2023-12-15T22:59:40.435038+0000","last_became_active":"2023-12-15T22:56:39.572176+0000","last_became_peered":"2023-12-15T22:56:39.572176+0000","last_unstale":"2023-12-15T22:59:40.435038+0000","last_undegraded":"2023-12-15T22:59:40.435038+0000","last_fullsized":"2023-12-15T22:59:40.435038+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7462374,"num_objects":13,"num_object_clones":9,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":57,"num_read_kb":29047,"num_write":60,"num_write_kb":18313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"192'139","reported_seq":457,"reported_epoch":193,"state":"active+clean","last_fresh":"2023-12-15T22:59:40.535406+0000","last_change":"2023-12-15T22:59:39.360045+0000","last_active":"2023-12-15T22:59:40.535406+0000","last_peered":"2023-12-15T22:59:40.535406+0000","last_clean":"2023-12-15T22:59:40.535406+0000","last_became_active":"2023-12-15T22:56:39.516361+0000","last_became_peered":"2023-12-15T22:56:39.516361+0000","last_unstale":"2023-12-15T22:59:40.535406+0000","last_undegraded":"2023-12-15T22:59:40.535406+0000","last_fullsized":"2023-12-15T22:59:40.535406+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":51,"num_read_kb":38717,"num_write":51,"num_write_kb":14458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"192'535","reported_seq":663,"reported_epoch":193,"state":"active+clean","last_fresh":"2023-12-15T22:59:40.538803+0000","last_change":"2023-12-15T22:59:40.541047+0000","last_active":"2023-12-15T22:59:40.538803+0000","last_peered":"2023-12-15T22:59:40.538803+0000","last_clean":"2023-12-15T22:59:40.538803+0000","last_became_active":"2023-12-15T22:56:39.519748+0000","last_became_peered":"2023-12-15T22:56:39.519748+0000","last_unstale":"2023-12-15T22:59:40.538803+0000","last_undegraded":"2023-12-15T22:59:40.538803+0000","last_fullsized":"2023-12-15T22:59:40.538803+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":535,"log_dups_size":0,"ondisk_log_size":535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12857487,"num_objects":48,"num_object_clones":38,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":198,"num_read_kb":121110,"num_write":200,"num_write_kb":57780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"184'127","reported_seq":456,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-12-15T22:59:38.339209+0000","last_change":"2023-12-15T22:59:38.328891+0000","last_active":"2023-12-15T22:59:38.339209+0000","last_peered":"2023-12-15T22:59:38.339209+0000","last_clean":"2023-12-15T22:59:38.339209+0000","last_became_active":"2023-12-15T22:56:39.543179+0000","last_became_peered":"2023-12-15T22:56:39.543179+0000","last_unstale":"2023-12-15T22:59:38.339209+0000","last_undegraded":"2023-12-15T22:59:38.339209+0000","last_fullsized":"2023-12-15T22:59:38.339209+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5949678,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":94,"num_read_kb":87445,"num_write":62,"num_write_kb":16612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":420,"reported_epoch":193,"state":"active+clean","last_fresh":"2023-12-15T22:59:40.345462+0000","last_change":"2023-12-15T22:56:32.607017+0000","last_active":"2023-12-15T22:59:40.345462+0000","last_peered":"2023-12-15T22:59:40.345462+0000","last_clean":"2023-12-15T22:59:40.345462+0000","last_became_active":"2023-12-15T22:56:32.602384+0000","last_became_peered":"2023-12-15T22:56:32.602384+0000","last_unstale":"2023-12-15T22:59:40.345462+0000","last_undegraded":"2023-12-15T22:59:40.345462+0000","last_fullsized":"2023-12-15T22:59:40.345462+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_clean_scrub_stamp":"2023-12-15T22:56:31.525572+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":277,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-12-15T22:59:38.326475+0000","last_change":"2023-12-15T22:56:33.627481+0000","last_active":"2023-12-15T22:59:38.326475+0000","last_peered":"2023-12-15T22:59:38.326475+0000","last_clean":"2023-12-15T22:59:38.326475+0000","last_became_active":"2023-12-15T22:56:33.624689+0000","last_became_peered":"2023-12-15T22:56:33.624689+0000","last_unstale":"2023-12-15T22:59:38.326475+0000","last_undegraded":"2023-12-15T22:59:38.326475+0000","last_fullsized":"2023-12-15T22:59:38.326475+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"184'158","reported_seq":491,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-12-15T22:59:38.320324+0000","last_change":"2023-12-15T22:59:38.326392+0000","last_active":"2023-12-15T22:59:38.320324+0000","last_peered":"2023-12-15T22:59:38.320324+0000","last_clean":"2023-12-15T22:59:38.320324+0000","last_became_active":"2023-12-15T22:56:39.529359+0000","last_became_peered":"2023-12-15T22:56:39.529359+0000","last_unstale":"2023-12-15T22:59:38.320324+0000","last_undegraded":"2023-12-15T22:59:38.320324+0000","last_fullsized":"2023-12-15T22:59:38.320324+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9210269,"num_objects":15,"num_object_clones":9,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":117,"num_read_kb":86105,"num_write":84,"num_write_kb":25166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":307,"reported_epoch":193,"state":"active+clean","last_fresh":"2023-12-15T22:59:40.565380+0000","last_change":"2023-12-15T22:56:33.655797+0000","last_active":"2023-12-15T22:59:40.565380+0000","last_peered":"2023-12-15T22:59:40.565380+0000","last_clean":"2023-12-15T22:59:40.565380+0000","last_became_active":"2023-12-15T22:56:33.652885+0000","last_became_peered":"2023-12-15T22:56:33.652885+0000","last_unstale":"2023-12-15T22:59:40.565380+0000","last_undegraded":"2023-12-15T22:59:40.565380+0000","last_fullsized":"2023-12-15T22:59:40.565380+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"184'240","reported_seq":523,"reported_epoch":193,"state":"active+clean","last_fresh":"2023-12-15T22:59:40.385423+0000","last_change":"2023-12-15T22:59:40.397409+0000","last_active":"2023-12-15T22:59:40.385423+0000","last_peered":"2023-12-15T22:59:40.385423+0000","last_clean":"2023-12-15T22:59:40.385423+0000","last_became_active":"2023-12-15T22:56:39.543509+0000","last_became_peered":"2023-12-15T22:56:39.543509+0000","last_unstale":"2023-12-15T22:59:40.385423+0000","last_undegraded":"2023-12-15T22:59:40.385423+0000","last_fullsized":"2023-12-15T22:59:40.385423+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":240,"log_dups_size":0,"ondisk_log_size":240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5748060,"num_objects":23,"num_object_clones":16,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":137,"num_read_kb":105605,"num_write":90,"num_write_kb":27441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":277,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-12-15T22:59:38.323240+0000","last_change":"2023-12-15T22:56:33.600485+0000","last_active":"2023-12-15T22:59:38.323240+0000","last_peered":"2023-12-15T22:59:38.323240+0000","last_clean":"2023-12-15T22:59:38.323240+0000","last_became_active":"2023-12-15T22:56:33.597320+0000","last_became_peered":"2023-12-15T22:56:33.597320+0000","last_unstale":"2023-12-15T22:59:38.323240+0000","last_undegraded":"2023-12-15T22:59:38.323240+0000","last_fullsized":"2023-12-15T22:59:38.323240+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"189'109","reported_seq":437,"reported_epoch":193,"state":"active+clean+snaptrim","last_fresh":"2023-12-15T22:59:40.349990+0000","last_change":"2023-12-15T22:59:40.349990+0000","last_active":"2023-12-15T22:59:40.349990+0000","last_peered":"2023-12-15T22:59:40.349990+0000","last_clean":"2023-12-15T22:59:40.349990+0000","last_became_active":"2023-12-15T22:56:39.560980+0000","last_became_peered":"2023-12-15T22:56:39.560980+0000","last_unstale":"2023-12-15T22:59:40.349990+0000","last_undegraded":"2023-12-15T22:59:40.349990+0000","last_fullsized":"2023-12-15T22:59:40.349990+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7462374,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":55,"num_read_kb":45341,"num_write":45,"num_write_kb":13643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":307,"reported_epoch":193,"state":"active+clean","last_fresh":"2023-12-15T22:59:40.536701+0000","last_change":"2023-12-15T22:56:33.648541+0000","last_active":"2023-12-15T22:59:40.536701+0000","last_peered":"2023-12-15T22:59:40.536701+0000","last_clean":"2023-12-15T22:59:40.536701+0000","last_became_active":"2023-12-15T22:56:33.643681+0000","last_became_peered":"2023-12-15T22:56:33.643681+0000","last_unstale":"2023-12-15T22:59:40.536701+0000","last_undegraded":"2023-12-15T22:59:40.536701+0000","last_fullsized":"2023-12-15T22:59:40.536701+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"184'183","reported_seq":472,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-12-15T22:59:37.656689+0000","last_change":"2023-12-15T22:59:36.656797+0000","last_active":"2023-12-15T22:59:37.656689+0000","last_peered":"2023-12-15T22:59:37.656689+0000","last_clean":"2023-12-15T22:59:37.656689+0000","last_became_active":"2023-12-15T22:56:39.506510+0000","last_became_peered":"2023-12-15T22:56:39.506510+0000","last_unstale":"2023-12-15T22:59:37.656689+0000","last_undegraded":"2023-12-15T22:59:37.656689+0000","last_fullsized":"2023-12-15T22:59:37.656689+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4773254,"num_objects":17,"num_object_clones":12,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":100,"num_read_kb":71478,"num_write":71,"num_write_kb":20602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"185'98","reported_seq":399,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-12-15T22:59:38.324126+0000","last_change":"2023-12-15T22:59:38.331291+0000","last_active":"2023-12-15T22:59:38.324126+0000","last_peered":"2023-12-15T22:59:38.324126+0000","last_clean":"2023-12-15T22:59:38.324126+0000","last_became_active":"2023-12-15T22:56:39.511159+0000","last_became_peered":"2023-12-15T22:56:39.511159+0000","last_unstale":"2023-12-15T22:59:38.324126+0000","last_undegraded":"2023-12-15T22:59:38.324126+0000","last_fullsized":"2023-12-15T22:59:38.324126+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":21,"num_read_kb":7340,"num_write":37,"num_write_kb":11102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":279,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-12-15T22:59:37.660143+0000","last_change":"2023-12-15T22:56:33.633890+0000","last_active":"2023-12-15T22:59:37.660143+0000","last_peered":"2023-12-15T22:59:37.660143+0000","last_clean":"2023-12-15T22:59:37.660143+0000","last_became_active":"2023-12-15T22:56:33.631134+0000","last_became_peered":"2023-12-15T22:56:33.631134+0000","last_unstale":"2023-12-15T22:59:37.660143+0000","last_undegraded":"2023-12-15T22:59:37.660143+0000","last_fullsized":"2023-12-15T22:59:37.660143+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"187'122","reported_seq":430,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-12-15T22:59:37.661623+0000","last_change":"2023-12-15T22:59:36.655658+0000","last_active":"2023-12-15T22:59:37.661623+0000","last_peered":"2023-12-15T22:59:37.661623+0000","last_clean":"2023-12-15T22:59:37.661623+0000","last_became_active":"2023-12-15T22:56:39.532730+0000","last_became_peered":"2023-12-15T22:56:39.532730+0000","last_unstale":"2023-12-15T22:59:37.661623+0000","last_undegraded":"2023-12-15T22:59:37.661623+0000","last_fullsized":"2023-12-15T22:59:37.661623+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3613538,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":57,"num_read_kb":44846,"num_write":49,"num_write_kb":14986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":291,"reported_epoch":193,"state":"active+clean","last_fresh":"2023-12-15T22:59:40.348115+0000","last_change":"2023-12-15T22:56:33.655736+0000","last_active":"2023-12-15T22:59:40.348115+0000","last_peered":"2023-12-15T22:59:40.348115+0000","last_clean":"2023-12-15T22:59:40.348115+0000","last_became_active":"2023-12-15T22:56:33.652917+0000","last_became_peered":"2023-12-15T22:56:33.652917+0000","last_unstale":"2023-12-15T22:59:40.348115+0000","last_undegraded":"2023-12-15T22:59:40.348115+0000","last_fullsized":"2023-12-15T22:59:40.348115+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"185'171","reported_seq":489,"reported_epoch":193,"state":"active+clean","last_fresh":"2023-12-15T22:59:40.353682+0000","last_change":"2023-12-15T22:59:40.358971+0000","last_active":"2023-12-15T22:59:40.353682+0000","last_peered":"2023-12-15T22:59:40.353682+0000","last_clean":"2023-12-15T22:59:40.353682+0000","last_became_active":"2023-12-15T22:56:39.557252+0000","last_became_peered":"2023-12-15T22:56:39.557252+0000","last_unstale":"2023-12-15T22:59:40.353682+0000","last_undegraded":"2023-12-15T22:59:40.353682+0000","last_fullsized":"2023-12-15T22:59:40.353682+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2353013,"num_objects":14,"num_object_clones":10,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":95,"num_read_kb":69776,"num_write":73,"num_write_kb":21141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":279,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-12-15T22:59:37.658764+0000","last_change":"2023-12-15T22:56:33.632348+0000","last_active":"2023-12-15T22:59:37.658764+0000","last_peered":"2023-12-15T22:59:37.658764+0000","last_clean":"2023-12-15T22:59:37.658764+0000","last_became_active":"2023-12-15T22:56:33.629022+0000","last_became_peered":"2023-12-15T22:56:33.629022+0000","last_unstale":"2023-12-15T22:59:37.658764+0000","last_undegraded":"2023-12-15T22:59:37.658764+0000","last_fullsized":"2023-12-15T22:59:37.658764+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"187'72","reported_seq":399,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-12-15T22:59:37.659430+0000","last_change":"2023-12-15T22:59:36.655976+0000","last_active":"2023-12-15T22:59:37.659430+0000","last_peered":"2023-12-15T22:59:37.659430+0000","last_clean":"2023-12-15T22:59:37.659430+0000","last_became_active":"2023-12-15T22:56:39.535145+0000","last_became_peered":"2023-12-15T22:56:39.535145+0000","last_unstale":"2023-12-15T22:59:37.659430+0000","last_undegraded":"2023-12-15T22:59:37.659430+0000","last_fullsized":"2023-12-15T22:59:37.659430+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3764801,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":31,"num_read_kb":29392,"num_write":27,"num_write_kb":8581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":277,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-12-15T22:59:38.323478+0000","last_change":"2023-12-15T22:56:33.627005+0000","last_active":"2023-12-15T22:59:38.323478+0000","last_peered":"2023-12-15T22:59:38.323478+0000","last_clean":"2023-12-15T22:59:38.323478+0000","last_became_active":"2023-12-15T22:56:33.622181+0000","last_became_peered":"2023-12-15T22:56:33.622181+0000","last_unstale":"2023-12-15T22:59:38.323478+0000","last_undegraded":"2023-12-15T22:59:38.323478+0000","last_fullsized":"2023-12-15T22:59:38.323478+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":90540035,"num_objects":250,"num_object_clones":183,"num_object_copies":750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":39,"num_read":1418,"num_read_kb":1069128,"num_write":1142,"num_write_kb":334700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2705,"ondisk_log_size":2705,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":79872,"kb_used_data":48888,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94291968,"statfs":{"total":96636764160,"available":96554975232,"internally_reserved":0,"allocated":50061312,"data_stored":50576592,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":69640,"kb_used_data":38656,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94302200,"statfs":{"total":96636764160,"available":96565452800,"internally_reserved":0,"allocated":39583744,"data_stored":41715737,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":72316,"kb_used_data":41332,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94299524,"statfs":{"total":96636764160,"available":96562712576,"internally_reserved":0,"allocated":42323968,"data_stored":42421129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":77584,"kb_used_data":42440,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94294256,"statfs":{"total":96636764160,"available":96557318144,"internally_reserved":0,"allocated":43458560,"data_stored":43304170,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-15T22:59:43.675 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T22:59:43.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:43.816+0000 7fa376e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:43.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:43.823+0000 7fa376e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:43.904 INFO:tasks.rados.rados.0.smithi071.stdout:749: seq_num 356 ranges {783292=540882,1932766=507787,3210350=572013} 2023-12-15T22:59:43.911 INFO:tasks.rados.rados.0.smithi071.stdout:749: writing smithi07183189-19 from 783292 to 1324174 tid 1 2023-12-15T22:59:43.917 INFO:tasks.rados.rados.0.smithi071.stdout:749: writing smithi07183189-19 from 1932766 to 2440553 tid 2 2023-12-15T22:59:43.925 INFO:tasks.rados.rados.0.smithi071.stdout:749: writing smithi07183189-19 from 3210350 to 3782363 tid 3 2023-12-15T22:59:43.925 INFO:tasks.rados.rados.0.smithi071.stdout:745: done (2 left) 2023-12-15T22:59:43.925 INFO:tasks.rados.rados.0.smithi071.stdout:747: done (1 left) 2023-12-15T22:59:43.925 INFO:tasks.rados.rados.0.smithi071.stdout:750: read oid 5 snap 89 2023-12-15T22:59:43.925 INFO:tasks.rados.rados.0.smithi071.stdout:750: expect deleted 2023-12-15T22:59:43.925 INFO:tasks.rados.rados.0.smithi071.stdout:751: write oid 25 current snap is 90 2023-12-15T22:59:43.925 INFO:tasks.rados.rados.0.smithi071.stdout:751: seq_num 357 ranges {400099=616131,1591248=408851} 2023-12-15T22:59:43.930 INFO:tasks.rados.rados.0.smithi071.stdout:751: writing smithi07183189-25 from 400099 to 1016230 tid 1 2023-12-15T22:59:43.936 INFO:tasks.rados.rados.0.smithi071.stdout:751: writing smithi07183189-25 from 1591248 to 2000099 tid 2 2023-12-15T22:59:43.936 INFO:tasks.rados.rados.0.smithi071.stdout:752: write (excl) oid 44 current snap is 90 2023-12-15T22:59:43.936 INFO:tasks.rados.rados.0.smithi071.stdout:752: seq_num 358 ranges {416906=407733,1366083=511456} 2023-12-15T22:59:43.940 INFO:tasks.rados.rados.0.smithi071.stdout:752: writing smithi07183189-44 from 416906 to 824639 tid 1 2023-12-15T22:59:43.946 INFO:tasks.rados.rados.0.smithi071.stdout:752: writing smithi07183189-44 from 1366083 to 1877539 tid 2 2023-12-15T22:59:43.946 INFO:tasks.rados.rados.0.smithi071.stdout:753: delete oid 22 current snap is 90 2023-12-15T22:59:43.947 INFO:tasks.rados.rados.0.smithi071.stdout:749: finishing write tid 1 to smithi07183189-19 2023-12-15T22:59:43.975 INFO:tasks.rados.rados.0.smithi071.stdout:751: finishing write tid 1 to smithi07183189-25 2023-12-15T22:59:44.001 INFO:tasks.rados.rados.0.smithi071.stdout:752: finishing write tid 1 to smithi07183189-44 2023-12-15T22:59:44.006 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:59:44.006 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-12-15T22:59:44.009 INFO:tasks.rados.rados.0.smithi071.stdout:749: finishing write tid 2 to smithi07183189-19 2023-12-15T22:59:44.019 INFO:tasks.rados.rados.0.smithi071.stdout:750: done (4 left) 2023-12-15T22:59:44.019 INFO:tasks.rados.rados.0.smithi071.stdout:753: done (3 left) 2023-12-15T22:59:44.019 INFO:tasks.rados.rados.0.smithi071.stdout:754: delete oid 13 current snap is 90 2023-12-15T22:59:44.020 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":286,"stamp":"2023-12-15T22:59:43.335781+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":91130403,"num_objects":252,"num_object_clones":183,"num_object_copies":756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":39,"num_read":1521,"num_read_kb":1069219,"num_write":1279,"num_write_kb":336594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2802,"ondisk_log_size":2802,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":299412,"kb_used_data":171316,"kb_used_omap":0,"kb_used_meta":128064,"kb_avail":377187948,"statfs":{"total":386547056640,"available":386240458752,"internally_reserved":0,"allocated":175427584,"data_stored":178017628,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":131137536},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5042166,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":12,"num_read_kb":8918,"num_write":26,"num_write_kb":8908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.018855"},"pg_stats":[{"pgid":"3.f","version":"185'90","reported_seq":453,"reported_epoch":193,"state":"active+clean","last_fresh":"2023-12-15T22:59:40.564752+0000","last_change":"2023-12-15T22:59:39.357157+0000","last_active":"2023-12-15T22:59:40.564752+0000","last_peered":"2023-12-15T22:59:40.564752+0000","last_clean":"2023-12-15T22:59:40.564752+0000","last_became_active":"2023-12-15T22:56:39.503797+0000","last_became_peered":"2023-12-15T22:56:39.503797+0000","last_unstale":"2023-12-15T22:59:40.564752+0000","last_undegraded":"2023-12-15T22:59:40.564752+0000","last_fullsized":"2023-12-15T22:59:40.564752+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6991778,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":70,"num_read_kb":53991,"num_write":36,"num_write_kb":10667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":360,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-12-15T22:59:37.663795+0000","last_change":"2023-12-15T22:59:36.656108+0000","last_active":"2023-12-15T22:59:37.663795+0000","last_peered":"2023-12-15T22:59:37.663795+0000","last_clean":"2023-12-15T22:59:37.663795+0000","last_became_active":"2023-12-15T22:56:39.552856+0000","last_became_peered":"2023-12-15T22:56:39.552856+0000","last_unstale":"2023-12-15T22:59:37.663795+0000","last_undegraded":"2023-12-15T22:59:37.663795+0000","last_fullsized":"2023-12-15T22:59:37.663795+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"185'263","reported_seq":571,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-12-15T22:59:38.325831+0000","last_change":"2023-12-15T22:59:38.330714+0000","last_active":"2023-12-15T22:59:38.325831+0000","last_peered":"2023-12-15T22:59:38.325831+0000","last_clean":"2023-12-15T22:59:38.325831+0000","last_became_active":"2023-12-15T22:56:39.495490+0000","last_became_peered":"2023-12-15T22:56:39.495490+0000","last_unstale":"2023-12-15T22:59:38.325831+0000","last_undegraded":"2023-12-15T22:59:38.325831+0000","last_fullsized":"2023-12-15T22:59:38.325831+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":263,"log_dups_size":0,"ondisk_log_size":263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10101040,"num_objects":27,"num_object_clones":18,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":199,"num_read_kb":180268,"num_write":132,"num_write_kb":38041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"192'257","reported_seq":558,"reported_epoch":193,"state":"active+clean+snaptrim","last_fresh":"2023-12-15T22:59:40.567511+0000","last_change":"2023-12-15T22:59:39.349202+0000","last_active":"2023-12-15T22:59:40.567511+0000","last_peered":"2023-12-15T22:59:40.567511+0000","last_clean":"2023-12-15T22:59:40.567511+0000","last_became_active":"2023-12-15T22:56:39.579234+0000","last_became_peered":"2023-12-15T22:56:39.579234+0000","last_unstale":"2023-12-15T22:59:40.567511+0000","last_undegraded":"2023-12-15T22:59:40.567511+0000","last_fullsized":"2023-12-15T22:59:40.567511+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10252369,"num_objects":24,"num_object_clones":20,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":136,"num_read_kb":98667,"num_write":125,"num_write_kb":36167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"193'141","reported_seq":457,"reported_epoch":193,"state":"active+clean","last_fresh":"2023-12-15T22:59:40.435038+0000","last_change":"2023-12-15T22:59:40.361645+0000","last_active":"2023-12-15T22:59:40.435038+0000","last_peered":"2023-12-15T22:59:40.435038+0000","last_clean":"2023-12-15T22:59:40.435038+0000","last_became_active":"2023-12-15T22:56:39.572176+0000","last_became_peered":"2023-12-15T22:56:39.572176+0000","last_unstale":"2023-12-15T22:59:40.435038+0000","last_undegraded":"2023-12-15T22:59:40.435038+0000","last_fullsized":"2023-12-15T22:59:40.435038+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7462374,"num_objects":13,"num_object_clones":9,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":57,"num_read_kb":29047,"num_write":60,"num_write_kb":18313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"192'139","reported_seq":457,"reported_epoch":193,"state":"active+clean","last_fresh":"2023-12-15T22:59:40.535406+0000","last_change":"2023-12-15T22:59:39.360045+0000","last_active":"2023-12-15T22:59:40.535406+0000","last_peered":"2023-12-15T22:59:40.535406+0000","last_clean":"2023-12-15T22:59:40.535406+0000","last_became_active":"2023-12-15T22:56:39.516361+0000","last_became_peered":"2023-12-15T22:56:39.516361+0000","last_unstale":"2023-12-15T22:59:40.535406+0000","last_undegraded":"2023-12-15T22:59:40.535406+0000","last_fullsized":"2023-12-15T22:59:40.535406+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":51,"num_read_kb":38717,"num_write":51,"num_write_kb":14458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"192'535","reported_seq":663,"reported_epoch":193,"state":"active+clean","last_fresh":"2023-12-15T22:59:40.538803+0000","last_change":"2023-12-15T22:59:40.541047+0000","last_active":"2023-12-15T22:59:40.538803+0000","last_peered":"2023-12-15T22:59:40.538803+0000","last_clean":"2023-12-15T22:59:40.538803+0000","last_became_active":"2023-12-15T22:56:39.519748+0000","last_became_peered":"2023-12-15T22:56:39.519748+0000","last_unstale":"2023-12-15T22:59:40.538803+0000","last_undegraded":"2023-12-15T22:59:40.538803+0000","last_fullsized":"2023-12-15T22:59:40.538803+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":535,"log_dups_size":0,"ondisk_log_size":535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12857487,"num_objects":48,"num_object_clones":38,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":198,"num_read_kb":121110,"num_write":200,"num_write_kb":57780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"184'127","reported_seq":456,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-12-15T22:59:38.339209+0000","last_change":"2023-12-15T22:59:38.328891+0000","last_active":"2023-12-15T22:59:38.339209+0000","last_peered":"2023-12-15T22:59:38.339209+0000","last_clean":"2023-12-15T22:59:38.339209+0000","last_became_active":"2023-12-15T22:56:39.543179+0000","last_became_peered":"2023-12-15T22:56:39.543179+0000","last_unstale":"2023-12-15T22:59:38.339209+0000","last_undegraded":"2023-12-15T22:59:38.339209+0000","last_fullsized":"2023-12-15T22:59:38.339209+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5949678,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":94,"num_read_kb":87445,"num_write":62,"num_write_kb":16612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":420,"reported_epoch":193,"state":"active+clean","last_fresh":"2023-12-15T22:59:40.345462+0000","last_change":"2023-12-15T22:56:32.607017+0000","last_active":"2023-12-15T22:59:40.345462+0000","last_peered":"2023-12-15T22:59:40.345462+0000","last_clean":"2023-12-15T22:59:40.345462+0000","last_became_active":"2023-12-15T22:56:32.602384+0000","last_became_peered":"2023-12-15T22:56:32.602384+0000","last_unstale":"2023-12-15T22:59:40.345462+0000","last_undegraded":"2023-12-15T22:59:40.345462+0000","last_fullsized":"2023-12-15T22:59:40.345462+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_clean_scrub_stamp":"2023-12-15T22:56:31.525572+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":277,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-12-15T22:59:38.326475+0000","last_change":"2023-12-15T22:56:33.627481+0000","last_active":"2023-12-15T22:59:38.326475+0000","last_peered":"2023-12-15T22:59:38.326475+0000","last_clean":"2023-12-15T22:59:38.326475+0000","last_became_active":"2023-12-15T22:56:33.624689+0000","last_became_peered":"2023-12-15T22:56:33.624689+0000","last_unstale":"2023-12-15T22:59:38.326475+0000","last_undegraded":"2023-12-15T22:59:38.326475+0000","last_fullsized":"2023-12-15T22:59:38.326475+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"184'158","reported_seq":491,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-12-15T22:59:38.320324+0000","last_change":"2023-12-15T22:59:38.326392+0000","last_active":"2023-12-15T22:59:38.320324+0000","last_peered":"2023-12-15T22:59:38.320324+0000","last_clean":"2023-12-15T22:59:38.320324+0000","last_became_active":"2023-12-15T22:56:39.529359+0000","last_became_peered":"2023-12-15T22:56:39.529359+0000","last_unstale":"2023-12-15T22:59:38.320324+0000","last_undegraded":"2023-12-15T22:59:38.320324+0000","last_fullsized":"2023-12-15T22:59:38.320324+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9210269,"num_objects":15,"num_object_clones":9,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":117,"num_read_kb":86105,"num_write":84,"num_write_kb":25166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":307,"reported_epoch":193,"state":"active+clean","last_fresh":"2023-12-15T22:59:40.565380+0000","last_change":"2023-12-15T22:56:33.655797+0000","last_active":"2023-12-15T22:59:40.565380+0000","last_peered":"2023-12-15T22:59:40.565380+0000","last_clean":"2023-12-15T22:59:40.565380+0000","last_became_active":"2023-12-15T22:56:33.652885+0000","last_became_peered":"2023-12-15T22:56:33.652885+0000","last_unstale":"2023-12-15T22:59:40.565380+0000","last_undegraded":"2023-12-15T22:59:40.565380+0000","last_fullsized":"2023-12-15T22:59:40.565380+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"184'240","reported_seq":523,"reported_epoch":193,"state":"active+clean","last_fresh":"2023-12-15T22:59:40.385423+0000","last_change":"2023-12-15T22:59:40.397409+0000","last_active":"2023-12-15T22:59:40.385423+0000","last_peered":"2023-12-15T22:59:40.385423+0000","last_clean":"2023-12-15T22:59:40.385423+0000","last_became_active":"2023-12-15T22:56:39.543509+0000","last_became_peered":"2023-12-15T22:56:39.543509+0000","last_unstale":"2023-12-15T22:59:40.385423+0000","last_undegraded":"2023-12-15T22:59:40.385423+0000","last_fullsized":"2023-12-15T22:59:40.385423+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":240,"log_dups_size":0,"ondisk_log_size":240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5748060,"num_objects":23,"num_object_clones":16,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":137,"num_read_kb":105605,"num_write":90,"num_write_kb":27441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":277,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-12-15T22:59:38.323240+0000","last_change":"2023-12-15T22:56:33.600485+0000","last_active":"2023-12-15T22:59:38.323240+0000","last_peered":"2023-12-15T22:59:38.323240+0000","last_clean":"2023-12-15T22:59:38.323240+0000","last_became_active":"2023-12-15T22:56:33.597320+0000","last_became_peered":"2023-12-15T22:56:33.597320+0000","last_unstale":"2023-12-15T22:59:38.323240+0000","last_undegraded":"2023-12-15T22:59:38.323240+0000","last_fullsized":"2023-12-15T22:59:38.323240+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"189'109","reported_seq":437,"reported_epoch":193,"state":"active+clean+snaptrim","last_fresh":"2023-12-15T22:59:40.349990+0000","last_change":"2023-12-15T22:59:40.349990+0000","last_active":"2023-12-15T22:59:40.349990+0000","last_peered":"2023-12-15T22:59:40.349990+0000","last_clean":"2023-12-15T22:59:40.349990+0000","last_became_active":"2023-12-15T22:56:39.560980+0000","last_became_peered":"2023-12-15T22:56:39.560980+0000","last_unstale":"2023-12-15T22:59:40.349990+0000","last_undegraded":"2023-12-15T22:59:40.349990+0000","last_fullsized":"2023-12-15T22:59:40.349990+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7462374,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":55,"num_read_kb":45341,"num_write":45,"num_write_kb":13643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":307,"reported_epoch":193,"state":"active+clean","last_fresh":"2023-12-15T22:59:40.536701+0000","last_change":"2023-12-15T22:56:33.648541+0000","last_active":"2023-12-15T22:59:40.536701+0000","last_peered":"2023-12-15T22:59:40.536701+0000","last_clean":"2023-12-15T22:59:40.536701+0000","last_became_active":"2023-12-15T22:56:33.643681+0000","last_became_peered":"2023-12-15T22:56:33.643681+0000","last_unstale":"2023-12-15T22:59:40.536701+0000","last_undegraded":"2023-12-15T22:59:40.536701+0000","last_fullsized":"2023-12-15T22:59:40.536701+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"184'183","reported_seq":472,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-12-15T22:59:37.656689+0000","last_change":"2023-12-15T22:59:36.656797+0000","last_active":"2023-12-15T22:59:37.656689+0000","last_peered":"2023-12-15T22:59:37.656689+0000","last_clean":"2023-12-15T22:59:37.656689+0000","last_became_active":"2023-12-15T22:56:39.506510+0000","last_became_peered":"2023-12-15T22:56:39.506510+0000","last_unstale":"2023-12-15T22:59:37.656689+0000","last_undegraded":"2023-12-15T22:59:37.656689+0000","last_fullsized":"2023-12-15T22:59:37.656689+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4773254,"num_objects":17,"num_object_clones":12,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":100,"num_read_kb":71478,"num_write":71,"num_write_kb":20602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"185'98","reported_seq":399,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-12-15T22:59:38.324126+0000","last_change":"2023-12-15T22:59:38.331291+0000","last_active":"2023-12-15T22:59:38.324126+0000","last_peered":"2023-12-15T22:59:38.324126+0000","last_clean":"2023-12-15T22:59:38.324126+0000","last_became_active":"2023-12-15T22:56:39.511159+0000","last_became_peered":"2023-12-15T22:56:39.511159+0000","last_unstale":"2023-12-15T22:59:38.324126+0000","last_undegraded":"2023-12-15T22:59:38.324126+0000","last_fullsized":"2023-12-15T22:59:38.324126+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":21,"num_read_kb":7340,"num_write":37,"num_write_kb":11102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":279,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-12-15T22:59:37.660143+0000","last_change":"2023-12-15T22:56:33.633890+0000","last_active":"2023-12-15T22:59:37.660143+0000","last_peered":"2023-12-15T22:59:37.660143+0000","last_clean":"2023-12-15T22:59:37.660143+0000","last_became_active":"2023-12-15T22:56:33.631134+0000","last_became_peered":"2023-12-15T22:56:33.631134+0000","last_unstale":"2023-12-15T22:59:37.660143+0000","last_undegraded":"2023-12-15T22:59:37.660143+0000","last_fullsized":"2023-12-15T22:59:37.660143+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"187'122","reported_seq":430,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-12-15T22:59:37.661623+0000","last_change":"2023-12-15T22:59:36.655658+0000","last_active":"2023-12-15T22:59:37.661623+0000","last_peered":"2023-12-15T22:59:37.661623+0000","last_clean":"2023-12-15T22:59:37.661623+0000","last_became_active":"2023-12-15T22:56:39.532730+0000","last_became_peered":"2023-12-15T22:56:39.532730+0000","last_unstale":"2023-12-15T22:59:37.661623+0000","last_undegraded":"2023-12-15T22:59:37.661623+0000","last_fullsized":"2023-12-15T22:59:37.661623+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3613538,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":57,"num_read_kb":44846,"num_write":49,"num_write_kb":14986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":291,"reported_epoch":193,"state":"active+clean","last_fresh":"2023-12-15T22:59:40.348115+0000","last_change":"2023-12-15T22:56:33.655736+0000","last_active":"2023-12-15T22:59:40.348115+0000","last_peered":"2023-12-15T22:59:40.348115+0000","last_clean":"2023-12-15T22:59:40.348115+0000","last_became_active":"2023-12-15T22:56:33.652917+0000","last_became_peered":"2023-12-15T22:56:33.652917+0000","last_unstale":"2023-12-15T22:59:40.348115+0000","last_undegraded":"2023-12-15T22:59:40.348115+0000","last_fullsized":"2023-12-15T22:59:40.348115+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"185'171","reported_seq":489,"reported_epoch":193,"state":"active+clean","last_fresh":"2023-12-15T22:59:40.353682+0000","last_change":"2023-12-15T22:59:40.358971+0000","last_active":"2023-12-15T22:59:40.353682+0000","last_peered":"2023-12-15T22:59:40.353682+0000","last_clean":"2023-12-15T22:59:40.353682+0000","last_became_active":"2023-12-15T22:56:39.557252+0000","last_became_peered":"2023-12-15T22:56:39.557252+0000","last_unstale":"2023-12-15T22:59:40.353682+0000","last_undegraded":"2023-12-15T22:59:40.353682+0000","last_fullsized":"2023-12-15T22:59:40.353682+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2353013,"num_objects":14,"num_object_clones":10,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":95,"num_read_kb":69776,"num_write":73,"num_write_kb":21141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":279,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-12-15T22:59:37.658764+0000","last_change":"2023-12-15T22:56:33.632348+0000","last_active":"2023-12-15T22:59:37.658764+0000","last_peered":"2023-12-15T22:59:37.658764+0000","last_clean":"2023-12-15T22:59:37.658764+0000","last_became_active":"2023-12-15T22:56:33.629022+0000","last_became_peered":"2023-12-15T22:56:33.629022+0000","last_unstale":"2023-12-15T22:59:37.658764+0000","last_undegraded":"2023-12-15T22:59:37.658764+0000","last_fullsized":"2023-12-15T22:59:37.658764+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"187'72","reported_seq":399,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-12-15T22:59:37.659430+0000","last_change":"2023-12-15T22:59:36.655976+0000","last_active":"2023-12-15T22:59:37.659430+0000","last_peered":"2023-12-15T22:59:37.659430+0000","last_clean":"2023-12-15T22:59:37.659430+0000","last_became_active":"2023-12-15T22:56:39.535145+0000","last_became_peered":"2023-12-15T22:56:39.535145+0000","last_unstale":"2023-12-15T22:59:37.659430+0000","last_undegraded":"2023-12-15T22:59:37.659430+0000","last_fullsized":"2023-12-15T22:59:37.659430+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3764801,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":31,"num_read_kb":29392,"num_write":27,"num_write_kb":8581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":277,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-12-15T22:59:38.323478+0000","last_change":"2023-12-15T22:56:33.627005+0000","last_active":"2023-12-15T22:59:38.323478+0000","last_peered":"2023-12-15T22:59:38.323478+0000","last_clean":"2023-12-15T22:59:38.323478+0000","last_became_active":"2023-12-15T22:56:33.622181+0000","last_became_peered":"2023-12-15T22:56:33.622181+0000","last_unstale":"2023-12-15T22:59:38.323478+0000","last_undegraded":"2023-12-15T22:59:38.323478+0000","last_fullsized":"2023-12-15T22:59:38.323478+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":90540035,"num_objects":250,"num_object_clones":183,"num_object_copies":750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":39,"num_read":1418,"num_read_kb":1069128,"num_write":1142,"num_write_kb":334700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2705,"ondisk_log_size":2705,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":79872,"kb_used_data":48888,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94291968,"statfs":{"total":96636764160,"available":96554975232,"internally_reserved":0,"allocated":50061312,"data_stored":50576592,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":69640,"kb_used_data":38656,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94302200,"statfs":{"total":96636764160,"available":96565452800,"internally_reserved":0,"allocated":39583744,"data_stored":41715737,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":72316,"kb_used_data":41332,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94299524,"statfs":{"total":96636764160,"available":96562712576,"internally_reserved":0,"allocated":42323968,"data_stored":42421129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":77584,"kb_used_data":42440,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94294256,"statfs":{"total":96636764160,"available":96557318144,"internally_reserved":0,"allocated":43458560,"data_stored":43304170,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-15T22:59:44.022 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2023-12-15T22:59:44.022 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T22:59:44.022 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-12-15T22:59:44.022 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-12-15T22:59:44.023 INFO:tasks.rados.rados.0.smithi071.stdout:751: finishing write tid 2 to smithi07183189-25 2023-12-15T22:59:44.027 INFO:tasks.rados.rados.0.smithi071.stdout:749: finishing write tid 3 to smithi07183189-19 2023-12-15T22:59:44.034 INFO:tasks.rados.rados.0.smithi071.stdout:752: finishing write tid 2 to smithi07183189-44 2023-12-15T22:59:44.055 INFO:tasks.rados.rados.0.smithi071.stdout:751: finishing write tid 3 to smithi07183189-25 2023-12-15T22:59:44.056 INFO:tasks.rados.rados.0.smithi071.stdout:751: finishing write tid 4 to smithi07183189-25 2023-12-15T22:59:44.057 INFO:tasks.rados.rados.0.smithi071.stdout:751: oid 25 updating version 0 to 266 2023-12-15T22:59:44.057 INFO:tasks.rados.rados.0.smithi071.stdout:751: oid 25 version 266 is already newer than 265 2023-12-15T22:59:44.057 INFO:tasks.rados.rados.0.smithi071.stdout:751: oid 25 updating version 266 to 267 2023-12-15T22:59:44.057 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 25 v 267 (ObjNum 357 snap 90 seq_num 357) dirty exists 2023-12-15T22:59:44.057 INFO:tasks.rados.rados.0.smithi071.stdout:751: left oid 25 (ObjNum 357 snap 90 seq_num 357) 2023-12-15T22:59:44.058 INFO:tasks.rados.rados.0.smithi071.stdout:752: finishing write tid 3 to smithi07183189-44 2023-12-15T22:59:44.059 INFO:tasks.rados.rados.0.smithi071.stdout:752: finishing write tid 4 to smithi07183189-44 2023-12-15T22:59:44.059 INFO:tasks.rados.rados.0.smithi071.stdout:752: oid 44 updating version 0 to 124 2023-12-15T22:59:44.059 INFO:tasks.rados.rados.0.smithi071.stdout:752: oid 44 updating version 124 to 125 2023-12-15T22:59:44.059 INFO:tasks.rados.rados.0.smithi071.stdout:752: oid 44 updating version 125 to 126 2023-12-15T22:59:44.059 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 126 (ObjNum 358 snap 90 seq_num 358) dirty exists 2023-12-15T22:59:44.059 INFO:tasks.rados.rados.0.smithi071.stdout:752: left oid 44 (ObjNum 358 snap 90 seq_num 358) 2023-12-15T22:59:44.063 INFO:tasks.rados.rados.0.smithi071.stdout:751: done (3 left) 2023-12-15T22:59:44.063 INFO:tasks.rados.rados.0.smithi071.stdout:752: done (2 left) 2023-12-15T22:59:44.063 INFO:tasks.rados.rados.0.smithi071.stdout:754: done (1 left) 2023-12-15T22:59:44.063 INFO:tasks.rados.rados.0.smithi071.stdout:755: snap_create 2023-12-15T22:59:44.064 INFO:tasks.rados.rados.0.smithi071.stdout:749: finishing write tid 4 to smithi07183189-19 2023-12-15T22:59:44.064 INFO:tasks.rados.rados.0.smithi071.stdout:749: finishing write tid 5 to smithi07183189-19 2023-12-15T22:59:44.065 INFO:tasks.rados.rados.0.smithi071.stdout:749: oid 19 updating version 0 to 266 2023-12-15T22:59:44.065 INFO:tasks.rados.rados.0.smithi071.stdout:749: oid 19 version 266 is already newer than 266 2023-12-15T22:59:44.065 INFO:tasks.rados.rados.0.smithi071.stdout:749: oid 19 updating version 266 to 267 2023-12-15T22:59:44.065 INFO:tasks.rados.rados.0.smithi071.stdout:749: oid 19 version 267 is already newer than 264 2023-12-15T22:59:44.065 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 267 (ObjNum 356 snap 90 seq_num 356) dirty exists 2023-12-15T22:59:44.065 INFO:tasks.rados.rados.0.smithi071.stdout:749: left oid 19 (ObjNum 356 snap 90 seq_num 356) 2023-12-15T22:59:44.130 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:44.126+0000 7fa6bbc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:44.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:44.134+0000 7fa6bbc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:44.146 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:59:44.146 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:59:44.146 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:59:44.146 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:59:44.146 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:59:44.146 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:59:44.146 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:59:44.146 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:59:44.146 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:59:44.146 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:59:44.147 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:59:44.147 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:59:44.148 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:59:44.149 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:59:44.149 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-12-15T22:59:44.159 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:59:44.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:44.293+0000 7fa75d6f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:44.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:44.297+0000 7fa75d6f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:44.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:44.293+0000 7f1a99b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:44.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:44.301+0000 7f1a99b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:44.314 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:59:44.314 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:59:44.314 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:59:44.314 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:59:44.314 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:59:44.314 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:59:44.314 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:59:44.315 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:59:44.315 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:59:44.315 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:59:44.315 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:59:44.315 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:59:44.317 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:59:44.317 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:59:44.318 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-12-15T22:59:44.336 INFO:tasks.rados.rados.0.smithi071.stdout:749: done (1 left) 2023-12-15T22:59:44.336 INFO:tasks.rados.rados.0.smithi071.stdout:755: done (0 left) 2023-12-15T22:59:44.336 INFO:tasks.rados.rados.0.smithi071.stdout:756: snap_remove snap 88 2023-12-15T22:59:44.426 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:44.422+0000 7f24ce06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:44.429 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:44.429+0000 7f24ce06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:44.441 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:59:44.442 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:59:44.442 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:59:44.442 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:59:44.442 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:59:44.442 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:59:44.442 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:59:44.442 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:59:44.442 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:59:44.442 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:59:44.442 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:59:44.443 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:59:44.444 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:59:44.445 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:59:44.445 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-12-15T22:59:44.551 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:44.547+0000 7f7d7062d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:44.555 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:44.554+0000 7f7d6f3cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:44.566 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:59:44.567 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:59:44.567 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:59:44.567 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:59:44.567 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:59:44.567 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:59:44.567 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:59:44.567 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:59:44.567 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:59:44.567 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:59:44.567 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:59:44.567 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:59:44.570 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:59:44.570 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:59:45.337 INFO:tasks.rados.rados.0.smithi071.stdout:756: done (0 left) 2023-12-15T22:59:45.337 INFO:tasks.rados.rados.0.smithi071.stdout:757: snap_remove snap 90 2023-12-15T22:59:46.341 INFO:tasks.rados.rados.0.smithi071.stdout:757: done (0 left) 2023-12-15T22:59:46.341 INFO:tasks.rados.rados.0.smithi071.stdout:758: write oid 26 current snap is 91 2023-12-15T22:59:46.341 INFO:tasks.rados.rados.0.smithi071.stdout:758: seq_num 361 ranges {467327=633480,1825176=242151} 2023-12-15T22:59:46.348 INFO:tasks.rados.rados.0.smithi071.stdout:758: writing smithi07183189-26 from 467327 to 1100807 tid 1 2023-12-15T22:59:46.354 INFO:tasks.rados.rados.0.smithi071.stdout:758: writing smithi07183189-26 from 1825176 to 2067327 tid 2 2023-12-15T22:59:46.355 INFO:tasks.rados.rados.0.smithi071.stdout:759: read oid 29 snap -1 2023-12-15T22:59:46.355 INFO:tasks.rados.rados.0.smithi071.stdout:759: expect (ObjNum 293 snap 80 seq_num 293) 2023-12-15T22:59:46.355 INFO:tasks.rados.rados.0.smithi071.stdout:760: write (excl) oid 9 current snap is 91 2023-12-15T22:59:46.355 INFO:tasks.rados.rados.0.smithi071.stdout:760: seq_num 362 ranges {484134=708729,1883658=200476} 2023-12-15T22:59:46.361 INFO:tasks.rados.rados.0.smithi071.stdout:760: writing smithi07183189-9 from 484134 to 1192863 tid 1 2023-12-15T22:59:46.367 INFO:tasks.rados.rados.0.smithi071.stdout:760: writing smithi07183189-9 from 1883658 to 2084134 tid 2 2023-12-15T22:59:46.367 INFO:tasks.rados.rados.0.smithi071.stdout:761: snap_remove snap 89 2023-12-15T22:59:46.386 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:59:46.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:46.500+0000 7f3b92d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:46.504 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:46.504+0000 7f3b92d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:47.266 INFO:tasks.rados.rados.0.smithi071.stdout:761: done (3 left) 2023-12-15T22:59:47.266 INFO:tasks.rados.rados.0.smithi071.stdout:762: delete oid 5 current snap is 91 2023-12-15T22:59:47.266 INFO:tasks.rados.rados.0.smithi071.stdout:759: expect (ObjNum 293 snap 80 seq_num 293) 2023-12-15T22:59:47.432 INFO:tasks.rados.rados.0.smithi071.stdout:758: finishing write tid 1 to smithi07183189-26 2023-12-15T22:59:47.432 INFO:tasks.rados.rados.0.smithi071.stdout:758: finishing write tid 2 to smithi07183189-26 2023-12-15T22:59:47.432 INFO:tasks.rados.rados.0.smithi071.stdout:760: finishing write tid 1 to smithi07183189-9 2023-12-15T22:59:47.432 INFO:tasks.rados.rados.0.smithi071.stdout:759: done (3 left) 2023-12-15T22:59:47.432 INFO:tasks.rados.rados.0.smithi071.stdout:762: done (2 left) 2023-12-15T22:59:47.432 INFO:tasks.rados.rados.0.smithi071.stdout:763: write (excl) oid 40 current snap is 91 2023-12-15T22:59:47.432 INFO:tasks.rados.rados.0.smithi071.stdout:758: finishing write tid 3 to smithi07183189-26 2023-12-15T22:59:47.432 INFO:tasks.rados.rados.0.smithi071.stdout:760: finishing write tid 2 to smithi07183189-9 2023-12-15T22:59:47.432 INFO:tasks.rados.rados.0.smithi071.stdout:763: seq_num 364 ranges {517748=459227,1600622=406110} 2023-12-15T22:59:47.437 INFO:tasks.rados.rados.0.smithi071.stdout:763: writing smithi07183189-40 from 517748 to 976975 tid 1 2023-12-15T22:59:47.444 INFO:tasks.rados.rados.0.smithi071.stdout:763: writing smithi07183189-40 from 1600622 to 2006732 tid 2 2023-12-15T22:59:47.444 INFO:tasks.rados.rados.0.smithi071.stdout:764: write oid 41 current snap is 91 2023-12-15T22:59:47.444 INFO:tasks.rados.rados.0.smithi071.stdout:758: finishing write tid 4 to smithi07183189-26 2023-12-15T22:59:47.444 INFO:tasks.rados.rados.0.smithi071.stdout:758: oid 26 updating version 0 to 175 2023-12-15T22:59:47.444 INFO:tasks.rados.rados.0.smithi071.stdout:758: oid 26 updating version 175 to 176 2023-12-15T22:59:47.444 INFO:tasks.rados.rados.0.smithi071.stdout:758: oid 26 version 176 is already newer than 174 2023-12-15T22:59:47.444 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 176 (ObjNum 361 snap 91 seq_num 361) dirty exists 2023-12-15T22:59:47.444 INFO:tasks.rados.rados.0.smithi071.stdout:758: left oid 26 (ObjNum 361 snap 91 seq_num 361) 2023-12-15T22:59:47.444 INFO:tasks.rados.rados.0.smithi071.stdout:760: finishing write tid 3 to smithi07183189-9 2023-12-15T22:59:47.444 INFO:tasks.rados.rados.0.smithi071.stdout:760: finishing write tid 4 to smithi07183189-9 2023-12-15T22:59:47.445 INFO:tasks.rados.rados.0.smithi071.stdout:760: oid 9 updating version 0 to 274 2023-12-15T22:59:47.445 INFO:tasks.rados.rados.0.smithi071.stdout:760: oid 9 updating version 274 to 275 2023-12-15T22:59:47.445 INFO:tasks.rados.rados.0.smithi071.stdout:760: oid 9 version 275 is already newer than 273 2023-12-15T22:59:47.445 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 275 (ObjNum 362 snap 91 seq_num 362) dirty exists 2023-12-15T22:59:47.445 INFO:tasks.rados.rados.0.smithi071.stdout:760: left oid 9 (ObjNum 362 snap 91 seq_num 362) 2023-12-15T22:59:47.445 INFO:tasks.rados.rados.0.smithi071.stdout:764: seq_num 365 ranges {534555=650829,1659104=475451} 2023-12-15T22:59:47.452 INFO:tasks.rados.rados.0.smithi071.stdout:764: writing smithi07183189-41 from 534555 to 1185384 tid 1 2023-12-15T22:59:47.459 INFO:tasks.rados.rados.0.smithi071.stdout:764: writing smithi07183189-41 from 1659104 to 2134555 tid 2 2023-12-15T22:59:47.459 INFO:tasks.rados.rados.0.smithi071.stdout:758: done (3 left) 2023-12-15T22:59:47.459 INFO:tasks.rados.rados.0.smithi071.stdout:760: done (2 left) 2023-12-15T22:59:47.459 INFO:tasks.rados.rados.0.smithi071.stdout:765: write (excl) oid 3 current snap is 91 2023-12-15T22:59:47.459 INFO:tasks.rados.rados.0.smithi071.stdout:763: finishing write tid 1 to smithi07183189-40 2023-12-15T22:59:47.459 INFO:tasks.rados.rados.0.smithi071.stdout:765: seq_num 366 ranges {551362=726078,1717586=409779} 2023-12-15T22:59:47.469 INFO:tasks.rados.rados.0.smithi071.stdout:765: writing smithi07183189-3 from 551362 to 1277440 tid 1 2023-12-15T22:59:47.475 INFO:tasks.rados.rados.0.smithi071.stdout:765: writing smithi07183189-3 from 1717586 to 2127365 tid 2 2023-12-15T22:59:47.475 INFO:tasks.rados.rados.0.smithi071.stdout:766: write (excl) oid 33 current snap is 91 2023-12-15T22:59:47.475 INFO:tasks.rados.rados.0.smithi071.stdout:766: seq_num 367 ranges {568169=401327,1376068=669790} 2023-12-15T22:59:47.481 INFO:tasks.rados.rados.0.smithi071.stdout:766: writing smithi07183189-33 from 568169 to 969496 tid 1 2023-12-15T22:59:47.490 INFO:tasks.rados.rados.0.smithi071.stdout:766: writing smithi07183189-33 from 1376068 to 2045858 tid 2 2023-12-15T22:59:47.490 INFO:tasks.rados.rados.0.smithi071.stdout:767: write (excl) oid 10 current snap is 91 2023-12-15T22:59:47.490 INFO:tasks.rados.rados.0.smithi071.stdout:763: finishing write tid 2 to smithi07183189-40 2023-12-15T22:59:47.490 INFO:tasks.rados.rados.0.smithi071.stdout:767: seq_num 368 ranges {584976=476576,1834550=350426} 2023-12-15T22:59:47.497 INFO:tasks.rados.rados.0.smithi071.stdout:767: writing smithi07183189-10 from 584976 to 1061552 tid 1 2023-12-15T22:59:47.505 INFO:tasks.rados.rados.0.smithi071.stdout:767: writing smithi07183189-10 from 1834550 to 2184976 tid 2 2023-12-15T22:59:47.505 INFO:tasks.rados.rados.0.smithi071.stdout:768: write (excl) oid 15 current snap is 91 2023-12-15T22:59:47.505 INFO:tasks.rados.rados.0.smithi071.stdout:768: seq_num 369 ranges {601783=551825,1776679=425104} 2023-12-15T22:59:47.513 INFO:tasks.rados.rados.0.smithi071.stdout:768: writing smithi07183189-15 from 601783 to 1153608 tid 1 2023-12-15T22:59:47.522 INFO:tasks.rados.rados.0.smithi071.stdout:768: writing smithi07183189-15 from 1776679 to 2201783 tid 2 2023-12-15T22:59:47.522 INFO:tasks.rados.rados.0.smithi071.stdout:769: write (excl) oid 7 current snap is 91 2023-12-15T22:59:47.522 INFO:tasks.rados.rados.0.smithi071.stdout:763: finishing write tid 3 to smithi07183189-40 2023-12-15T22:59:47.522 INFO:tasks.rados.rados.0.smithi071.stdout:763: finishing write tid 4 to smithi07183189-40 2023-12-15T22:59:47.522 INFO:tasks.rados.rados.0.smithi071.stdout:763: oid 40 updating version 0 to 543 2023-12-15T22:59:47.522 INFO:tasks.rados.rados.0.smithi071.stdout:763: oid 40 version 543 is already newer than 542 2023-12-15T22:59:47.522 INFO:tasks.rados.rados.0.smithi071.stdout:763: oid 40 version 543 is already newer than 541 2023-12-15T22:59:47.522 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 40 v 543 (ObjNum 364 snap 91 seq_num 364) dirty exists 2023-12-15T22:59:47.522 INFO:tasks.rados.rados.0.smithi071.stdout:763: left oid 40 (ObjNum 364 snap 91 seq_num 364) 2023-12-15T22:59:47.522 INFO:tasks.rados.rados.0.smithi071.stdout:769: seq_num 370 ranges {618590=627074,1835161=383429} 2023-12-15T22:59:47.531 INFO:tasks.rados.rados.0.smithi071.stdout:769: writing smithi07183189-7 from 618590 to 1245664 tid 1 2023-12-15T22:59:47.538 INFO:tasks.rados.rados.0.smithi071.stdout:769: writing smithi07183189-7 from 1835161 to 2218590 tid 2 2023-12-15T22:59:47.538 INFO:tasks.rados.rados.0.smithi071.stdout:763: done (6 left) 2023-12-15T22:59:47.538 INFO:tasks.rados.rados.0.smithi071.stdout:770: read oid 11 snap -1 2023-12-15T22:59:47.538 INFO:tasks.rados.rados.0.smithi071.stdout:770: expect (ObjNum 323 snap 81 seq_num 323) 2023-12-15T22:59:47.538 INFO:tasks.rados.rados.0.smithi071.stdout:771: write oid 4 current snap is 91 2023-12-15T22:59:47.538 INFO:tasks.rados.rados.0.smithi071.stdout:771: seq_num 371 ranges {635397=702323,1893643=341754} 2023-12-15T22:59:47.547 INFO:tasks.rados.rados.0.smithi071.stdout:771: writing smithi07183189-4 from 635397 to 1337720 tid 1 2023-12-15T22:59:47.554 INFO:tasks.rados.rados.0.smithi071.stdout:771: writing smithi07183189-4 from 1893643 to 2235397 tid 2 2023-12-15T22:59:47.554 INFO:tasks.rados.rados.0.smithi071.stdout:772: read oid 46 snap -1 2023-12-15T22:59:47.554 INFO:tasks.rados.rados.0.smithi071.stdout:772: expect deleted 2023-12-15T22:59:47.554 INFO:tasks.rados.rados.0.smithi071.stdout:773: read oid 18 snap -1 2023-12-15T22:59:47.554 INFO:tasks.rados.rados.0.smithi071.stdout:773: expect deleted 2023-12-15T22:59:47.554 INFO:tasks.rados.rados.0.smithi071.stdout:774: read oid 42 snap -1 2023-12-15T22:59:47.554 INFO:tasks.rados.rados.0.smithi071.stdout:774: expect deleted 2023-12-15T22:59:47.554 INFO:tasks.rados.rados.0.smithi071.stdout:775: read oid 49 snap -1 2023-12-15T22:59:47.554 INFO:tasks.rados.rados.0.smithi071.stdout:775: expect (ObjNum 246 snap 68 seq_num 246) 2023-12-15T22:59:47.555 INFO:tasks.rados.rados.0.smithi071.stdout:776: write oid 8 current snap is 91 2023-12-15T22:59:47.555 INFO:tasks.rados.rados.0.smithi071.stdout:776: seq_num 372 ranges {652204=777572,1952125=300079} 2023-12-15T22:59:47.564 INFO:tasks.rados.rados.0.smithi071.stdout:776: writing smithi07183189-8 from 652204 to 1429776 tid 1 2023-12-15T22:59:47.570 INFO:tasks.rados.rados.0.smithi071.stdout:776: writing smithi07183189-8 from 1952125 to 2252204 tid 2 2023-12-15T22:59:47.570 INFO:tasks.rados.rados.0.smithi071.stdout:777: write oid 48 current snap is 91 2023-12-15T22:59:47.570 INFO:tasks.rados.rados.0.smithi071.stdout:777: seq_num 373 ranges {669011=569174,2010607=258404} 2023-12-15T22:59:47.579 INFO:tasks.rados.rados.0.smithi071.stdout:777: writing smithi07183189-48 from 669011 to 1238185 tid 1 2023-12-15T22:59:47.586 INFO:tasks.rados.rados.0.smithi071.stdout:777: writing smithi07183189-48 from 2010607 to 2269011 tid 2 2023-12-15T22:59:47.586 INFO:tasks.rados.rados.0.smithi071.stdout:778: write (excl) oid 31 current snap is 91 2023-12-15T22:59:47.586 INFO:tasks.rados.rados.0.smithi071.stdout:764: finishing write tid 1 to smithi07183189-41 2023-12-15T22:59:47.586 INFO:tasks.rados.rados.0.smithi071.stdout:778: seq_num 374 ranges {685818=644423,2069089=216729} 2023-12-15T22:59:47.595 INFO:tasks.rados.rados.0.smithi071.stdout:778: writing smithi07183189-31 from 685818 to 1330241 tid 1 2023-12-15T22:59:47.602 INFO:tasks.rados.rados.0.smithi071.stdout:778: writing smithi07183189-31 from 2069089 to 2285818 tid 2 2023-12-15T22:59:47.602 INFO:tasks.rados.rados.0.smithi071.stdout:779: write (excl) oid 6 current snap is 91 2023-12-15T22:59:47.602 INFO:tasks.rados.rados.0.smithi071.stdout:765: finishing write tid 1 to smithi07183189-3 2023-12-15T22:59:47.602 INFO:tasks.rados.rados.0.smithi071.stdout:779: seq_num 375 ranges {702625=719672,2127571=175054} 2023-12-15T22:59:47.612 INFO:tasks.rados.rados.0.smithi071.stdout:779: writing smithi07183189-6 from 702625 to 1422297 tid 1 2023-12-15T22:59:47.618 INFO:tasks.rados.rados.0.smithi071.stdout:779: writing smithi07183189-6 from 2127571 to 2302625 tid 2 2023-12-15T22:59:47.618 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2023-12-15T22:59:47.618 INFO:tasks.rados.rados.0.smithi071.stdout:766: finishing write tid 1 to smithi07183189-33 2023-12-15T22:59:47.618 INFO:tasks.rados.rados.0.smithi071.stdout:767: finishing write tid 1 to smithi07183189-10 2023-12-15T22:59:47.618 INFO:tasks.rados.rados.0.smithi071.stdout:773: done (15 left) 2023-12-15T22:59:47.618 INFO:tasks.rados.rados.0.smithi071.stdout:780: write oid 30 current snap is 91 2023-12-15T22:59:47.618 INFO:tasks.rados.rados.0.smithi071.stdout:780: seq_num 376 ranges {719432=794921,2186053=133379} 2023-12-15T22:59:47.628 INFO:tasks.rados.rados.0.smithi071.stdout:780: writing smithi07183189-30 from 719432 to 1514353 tid 1 2023-12-15T22:59:47.634 INFO:tasks.rados.rados.0.smithi071.stdout:780: writing smithi07183189-30 from 2186053 to 2319432 tid 2 2023-12-15T22:59:47.634 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2023-12-15T22:59:47.634 INFO:tasks.rados.rados.0.smithi071.stdout:764: finishing write tid 2 to smithi07183189-41 2023-12-15T22:59:47.634 INFO:tasks.rados.rados.0.smithi071.stdout:765: finishing write tid 2 to smithi07183189-3 2023-12-15T22:59:47.634 INFO:tasks.rados.rados.0.smithi071.stdout:772: done (15 left) 2023-12-15T22:59:47.634 INFO:tasks.rados.rados.0.smithi071.stdout:781: delete oid 27 current snap is 91 2023-12-15T22:59:47.653 INFO:tasks.rados.rados.0.smithi071.stdout:775: expect (ObjNum 246 snap 68 seq_num 246) 2023-12-15T22:59:47.805 INFO:tasks.rados.rados.0.smithi071.stdout:767: finishing write tid 2 to smithi07183189-10 2023-12-15T22:59:47.805 INFO:tasks.rados.rados.0.smithi071.stdout:766: finishing write tid 2 to smithi07183189-33 2023-12-15T22:59:47.805 INFO:tasks.rados.rados.0.smithi071.stdout:765: finishing write tid 3 to smithi07183189-3 2023-12-15T22:59:47.805 INFO:tasks.rados.rados.0.smithi071.stdout:775: done (15 left) 2023-12-15T22:59:47.805 INFO:tasks.rados.rados.0.smithi071.stdout:781: done (14 left) 2023-12-15T22:59:47.805 INFO:tasks.rados.rados.0.smithi071.stdout:782: delete oid 19 current snap is 91 2023-12-15T22:59:47.805 INFO:tasks.rados.rados.0.smithi071.stdout:765: finishing write tid 4 to smithi07183189-3 2023-12-15T22:59:47.806 INFO:tasks.rados.rados.0.smithi071.stdout:765: oid 3 updating version 0 to 129 2023-12-15T22:59:47.806 INFO:tasks.rados.rados.0.smithi071.stdout:765: oid 3 updating version 129 to 130 2023-12-15T22:59:47.806 INFO:tasks.rados.rados.0.smithi071.stdout:765: oid 3 version 130 is already newer than 128 2023-12-15T22:59:47.806 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 130 (ObjNum 366 snap 91 seq_num 366) dirty exists 2023-12-15T22:59:47.806 INFO:tasks.rados.rados.0.smithi071.stdout:765: left oid 3 (ObjNum 366 snap 91 seq_num 366) 2023-12-15T22:59:47.806 INFO:tasks.rados.rados.0.smithi071.stdout:766: finishing write tid 3 to smithi07183189-33 2023-12-15T22:59:47.806 INFO:tasks.rados.rados.0.smithi071.stdout:764: finishing write tid 3 to smithi07183189-41 2023-12-15T22:59:47.806 INFO:tasks.rados.rados.0.smithi071.stdout:766: finishing write tid 4 to smithi07183189-33 2023-12-15T22:59:47.806 INFO:tasks.rados.rados.0.smithi071.stdout:766: oid 33 updating version 0 to 544 2023-12-15T22:59:47.806 INFO:tasks.rados.rados.0.smithi071.stdout:766: oid 33 updating version 544 to 545 2023-12-15T22:59:47.806 INFO:tasks.rados.rados.0.smithi071.stdout:766: oid 33 updating version 545 to 546 2023-12-15T22:59:47.806 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 33 v 546 (ObjNum 367 snap 91 seq_num 367) dirty exists 2023-12-15T22:59:47.806 INFO:tasks.rados.rados.0.smithi071.stdout:766: left oid 33 (ObjNum 367 snap 91 seq_num 367) 2023-12-15T22:59:47.806 INFO:tasks.rados.rados.0.smithi071.stdout:778: finishing write tid 1 to smithi07183189-31 2023-12-15T22:59:47.806 INFO:tasks.rados.rados.0.smithi071.stdout:769: finishing write tid 1 to smithi07183189-7 2023-12-15T22:59:47.806 INFO:tasks.rados.rados.0.smithi071.stdout:764: finishing write tid 4 to smithi07183189-41 2023-12-15T22:59:47.807 INFO:tasks.rados.rados.0.smithi071.stdout:764: oid 41 updating version 0 to 247 2023-12-15T22:59:47.807 INFO:tasks.rados.rados.0.smithi071.stdout:764: oid 41 version 247 is already newer than 246 2023-12-15T22:59:47.807 INFO:tasks.rados.rados.0.smithi071.stdout:764: oid 41 version 247 is already newer than 245 2023-12-15T22:59:47.807 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 41 v 247 (ObjNum 365 snap 91 seq_num 365) dirty exists 2023-12-15T22:59:47.807 INFO:tasks.rados.rados.0.smithi071.stdout:764: left oid 41 (ObjNum 365 snap 91 seq_num 365) 2023-12-15T22:59:47.807 INFO:tasks.rados.rados.0.smithi071.stdout:780: finishing write tid 1 to smithi07183189-30 2023-12-15T22:59:47.807 INFO:tasks.rados.rados.0.smithi071.stdout:767: finishing write tid 3 to smithi07183189-10 2023-12-15T22:59:47.807 INFO:tasks.rados.rados.0.smithi071.stdout:767: finishing write tid 4 to smithi07183189-10 2023-12-15T22:59:47.807 INFO:tasks.rados.rados.0.smithi071.stdout:767: oid 10 updating version 0 to 161 2023-12-15T22:59:47.807 INFO:tasks.rados.rados.0.smithi071.stdout:767: oid 10 version 161 is already newer than 159 2023-12-15T22:59:47.807 INFO:tasks.rados.rados.0.smithi071.stdout:767: oid 10 version 161 is already newer than 160 2023-12-15T22:59:47.807 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 10 v 161 (ObjNum 368 snap 91 seq_num 368) dirty exists 2023-12-15T22:59:47.807 INFO:tasks.rados.rados.0.smithi071.stdout:767: left oid 10 (ObjNum 368 snap 91 seq_num 368) 2023-12-15T22:59:47.807 INFO:tasks.rados.rados.0.smithi071.stdout:776: finishing write tid 1 to smithi07183189-8 2023-12-15T22:59:47.807 INFO:tasks.rados.rados.0.smithi071.stdout:769: finishing write tid 2 to smithi07183189-7 2023-12-15T22:59:47.807 INFO:tasks.rados.rados.0.smithi071.stdout:778: finishing write tid 2 to smithi07183189-31 2023-12-15T22:59:47.808 INFO:tasks.rados.rados.0.smithi071.stdout:780: finishing write tid 2 to smithi07183189-30 2023-12-15T22:59:47.808 INFO:tasks.rados.rados.0.smithi071.stdout:768: finishing write tid 1 to smithi07183189-15 2023-12-15T22:59:47.808 INFO:tasks.rados.rados.0.smithi071.stdout:776: finishing write tid 2 to smithi07183189-8 2023-12-15T22:59:47.808 INFO:tasks.rados.rados.0.smithi071.stdout:778: finishing write tid 3 to smithi07183189-31 2023-12-15T22:59:47.808 INFO:tasks.rados.rados.0.smithi071.stdout:769: finishing write tid 3 to smithi07183189-7 2023-12-15T22:59:47.808 INFO:tasks.rados.rados.0.smithi071.stdout:778: finishing write tid 4 to smithi07183189-31 2023-12-15T22:59:47.808 INFO:tasks.rados.rados.0.smithi071.stdout:778: oid 31 updating version 0 to 180 2023-12-15T22:59:47.808 INFO:tasks.rados.rados.0.smithi071.stdout:778: oid 31 updating version 180 to 181 2023-12-15T22:59:47.808 INFO:tasks.rados.rados.0.smithi071.stdout:778: oid 31 version 181 is already newer than 179 2023-12-15T22:59:47.808 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 181 (ObjNum 374 snap 91 seq_num 374) dirty exists 2023-12-15T22:59:47.808 INFO:tasks.rados.rados.0.smithi071.stdout:778: left oid 31 (ObjNum 374 snap 91 seq_num 374) 2023-12-15T22:59:47.808 INFO:tasks.rados.rados.0.smithi071.stdout:780: finishing write tid 3 to smithi07183189-30 2023-12-15T22:59:47.808 INFO:tasks.rados.rados.0.smithi071.stdout:780: finishing write tid 4 to smithi07183189-30 2023-12-15T22:59:47.808 INFO:tasks.rados.rados.0.smithi071.stdout:780: oid 30 updating version 0 to 119 2023-12-15T22:59:47.808 INFO:tasks.rados.rados.0.smithi071.stdout:780: oid 30 updating version 119 to 120 2023-12-15T22:59:47.809 INFO:tasks.rados.rados.0.smithi071.stdout:780: oid 30 version 120 is already newer than 118 2023-12-15T22:59:47.809 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 120 (ObjNum 376 snap 91 seq_num 376) dirty exists 2023-12-15T22:59:47.809 INFO:tasks.rados.rados.0.smithi071.stdout:780: left oid 30 (ObjNum 376 snap 91 seq_num 376) 2023-12-15T22:59:47.818 INFO:tasks.rados.rados.0.smithi071.stdout:779: finishing write tid 1 to smithi07183189-6 2023-12-15T22:59:47.818 INFO:tasks.rados.rados.0.smithi071.stdout:769: finishing write tid 4 to smithi07183189-7 2023-12-15T22:59:47.818 INFO:tasks.rados.rados.0.smithi071.stdout:769: oid 7 updating version 0 to 273 2023-12-15T22:59:47.818 INFO:tasks.rados.rados.0.smithi071.stdout:769: oid 7 updating version 273 to 274 2023-12-15T22:59:47.818 INFO:tasks.rados.rados.0.smithi071.stdout:769: oid 7 version 274 is already newer than 272 2023-12-15T22:59:47.818 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 274 (ObjNum 370 snap 91 seq_num 370) dirty exists 2023-12-15T22:59:47.818 INFO:tasks.rados.rados.0.smithi071.stdout:769: left oid 7 (ObjNum 370 snap 91 seq_num 370) 2023-12-15T22:59:47.832 INFO:tasks.rados.rados.0.smithi071.stdout:764: done (14 left) 2023-12-15T22:59:47.832 INFO:tasks.rados.rados.0.smithi071.stdout:765: done (13 left) 2023-12-15T22:59:47.832 INFO:tasks.rados.rados.0.smithi071.stdout:766: done (12 left) 2023-12-15T22:59:47.832 INFO:tasks.rados.rados.0.smithi071.stdout:767: done (11 left) 2023-12-15T22:59:47.832 INFO:tasks.rados.rados.0.smithi071.stdout:769: done (10 left) 2023-12-15T22:59:47.832 INFO:tasks.rados.rados.0.smithi071.stdout:778: done (9 left) 2023-12-15T22:59:47.832 INFO:tasks.rados.rados.0.smithi071.stdout:780: done (8 left) 2023-12-15T22:59:47.832 INFO:tasks.rados.rados.0.smithi071.stdout:782: done (7 left) 2023-12-15T22:59:47.832 INFO:tasks.rados.rados.0.smithi071.stdout:783: delete oid 2 current snap is 91 2023-12-15T22:59:47.833 INFO:tasks.rados.rados.0.smithi071.stdout:776: finishing write tid 3 to smithi07183189-8 2023-12-15T22:59:47.841 INFO:tasks.rados.rados.0.smithi071.stdout:768: finishing write tid 2 to smithi07183189-15 2023-12-15T22:59:47.841 INFO:tasks.rados.rados.0.smithi071.stdout:776: finishing write tid 4 to smithi07183189-8 2023-12-15T22:59:47.842 INFO:tasks.rados.rados.0.smithi071.stdout:776: oid 8 updating version 0 to 250 2023-12-15T22:59:47.842 INFO:tasks.rados.rados.0.smithi071.stdout:776: oid 8 updating version 250 to 252 2023-12-15T22:59:47.842 INFO:tasks.rados.rados.0.smithi071.stdout:776: oid 8 version 252 is already newer than 252 2023-12-15T22:59:47.842 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 8 v 252 (ObjNum 372 snap 91 seq_num 372) dirty exists 2023-12-15T22:59:47.842 INFO:tasks.rados.rados.0.smithi071.stdout:776: left oid 8 (ObjNum 372 snap 91 seq_num 372) 2023-12-15T22:59:47.842 INFO:tasks.rados.rados.0.smithi071.stdout:776: done (7 left) 2023-12-15T22:59:47.843 INFO:tasks.rados.rados.0.smithi071.stdout:783: done (6 left) 2023-12-15T22:59:47.843 INFO:tasks.rados.rados.0.smithi071.stdout:784: read oid 27 snap -1 2023-12-15T22:59:47.843 INFO:tasks.rados.rados.0.smithi071.stdout:784: expect deleted 2023-12-15T22:59:47.843 INFO:tasks.rados.rados.0.smithi071.stdout:785: write oid 31 current snap is 91 2023-12-15T22:59:47.843 INFO:tasks.rados.rados.0.smithi071.stdout:785: seq_num 380 ranges {786660=695917,1903628=435462} 2023-12-15T22:59:47.853 INFO:tasks.rados.rados.0.smithi071.stdout:785: writing smithi07183189-31 from 786660 to 1482577 tid 1 2023-12-15T22:59:47.860 INFO:tasks.rados.rados.0.smithi071.stdout:785: writing smithi07183189-31 from 1903628 to 2339090 tid 2 2023-12-15T22:59:47.860 INFO:tasks.rados.rados.0.smithi071.stdout:786: read oid 18 snap -1 2023-12-15T22:59:47.860 INFO:tasks.rados.rados.0.smithi071.stdout:786: expect deleted 2023-12-15T22:59:47.860 INFO:tasks.rados.rados.0.smithi071.stdout:787: snap_create 2023-12-15T22:59:47.860 INFO:tasks.rados.rados.0.smithi071.stdout:771: finishing write tid 1 to smithi07183189-4 2023-12-15T22:59:47.861 INFO:tasks.rados.rados.0.smithi071.stdout:779: finishing write tid 2 to smithi07183189-6 2023-12-15T22:59:47.861 INFO:tasks.rados.rados.0.smithi071.stdout:771: finishing write tid 2 to smithi07183189-4 2023-12-15T22:59:47.861 INFO:tasks.rados.rados.0.smithi071.stdout:771: finishing write tid 3 to smithi07183189-4 2023-12-15T22:59:47.868 INFO:tasks.rados.rados.0.smithi071.stdout:771: finishing write tid 4 to smithi07183189-4 2023-12-15T22:59:47.868 INFO:tasks.rados.rados.0.smithi071.stdout:771: oid 4 updating version 0 to 277 2023-12-15T22:59:47.868 INFO:tasks.rados.rados.0.smithi071.stdout:771: oid 4 version 277 is already newer than 277 2023-12-15T22:59:47.868 INFO:tasks.rados.rados.0.smithi071.stdout:771: oid 4 version 277 is already newer than 277 2023-12-15T22:59:47.868 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 4 v 277 (ObjNum 371 snap 91 seq_num 371) dirty exists 2023-12-15T22:59:47.868 INFO:tasks.rados.rados.0.smithi071.stdout:771: left oid 4 (ObjNum 371 snap 91 seq_num 371) 2023-12-15T22:59:47.884 INFO:tasks.rados.rados.0.smithi071.stdout:779: finishing write tid 3 to smithi07183189-6 2023-12-15T22:59:47.886 INFO:tasks.rados.rados.0.smithi071.stdout:768: finishing write tid 3 to smithi07183189-15 2023-12-15T22:59:47.891 INFO:tasks.rados.rados.0.smithi071.stdout:779: finishing write tid 4 to smithi07183189-6 2023-12-15T22:59:47.891 INFO:tasks.rados.rados.0.smithi071.stdout:779: oid 6 updating version 0 to 132 2023-12-15T22:59:47.891 INFO:tasks.rados.rados.0.smithi071.stdout:779: oid 6 version 132 is already newer than 131 2023-12-15T22:59:47.891 INFO:tasks.rados.rados.0.smithi071.stdout:779: oid 6 updating version 132 to 133 2023-12-15T22:59:47.892 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 6 v 133 (ObjNum 375 snap 91 seq_num 375) dirty exists 2023-12-15T22:59:47.892 INFO:tasks.rados.rados.0.smithi071.stdout:779: left oid 6 (ObjNum 375 snap 91 seq_num 375) 2023-12-15T22:59:47.892 INFO:tasks.rados.rados.0.smithi071.stdout:768: finishing write tid 4 to smithi07183189-15 2023-12-15T22:59:47.892 INFO:tasks.rados.rados.0.smithi071.stdout:768: oid 15 updating version 0 to 162 2023-12-15T22:59:47.892 INFO:tasks.rados.rados.0.smithi071.stdout:768: oid 15 updating version 162 to 164 2023-12-15T22:59:47.892 INFO:tasks.rados.rados.0.smithi071.stdout:768: oid 15 version 164 is already newer than 163 2023-12-15T22:59:47.892 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 164 (ObjNum 369 snap 91 seq_num 369) dirty exists 2023-12-15T22:59:47.892 INFO:tasks.rados.rados.0.smithi071.stdout:768: left oid 15 (ObjNum 369 snap 91 seq_num 369) 2023-12-15T22:59:47.892 INFO:tasks.rados.rados.0.smithi071.stdout:785: finishing write tid 1 to smithi07183189-31 2023-12-15T22:59:47.906 INFO:tasks.rados.rados.0.smithi071.stdout:770: expect (ObjNum 323 snap 81 seq_num 323) 2023-12-15T22:59:48.157 INFO:tasks.rados.rados.0.smithi071.stdout:777: finishing write tid 1 to smithi07183189-48 2023-12-15T22:59:48.157 INFO:tasks.rados.rados.0.smithi071.stdout:777: finishing write tid 2 to smithi07183189-48 2023-12-15T22:59:48.158 INFO:tasks.rados.rados.0.smithi071.stdout:785: finishing write tid 2 to smithi07183189-31 2023-12-15T22:59:48.158 INFO:tasks.rados.rados.0.smithi071.stdout:777: finishing write tid 3 to smithi07183189-48 2023-12-15T22:59:48.158 INFO:tasks.rados.rados.0.smithi071.stdout:777: finishing write tid 4 to smithi07183189-48 2023-12-15T22:59:48.158 INFO:tasks.rados.rados.0.smithi071.stdout:777: oid 48 updating version 0 to 167 2023-12-15T22:59:48.158 INFO:tasks.rados.rados.0.smithi071.stdout:777: oid 48 version 167 is already newer than 167 2023-12-15T22:59:48.158 INFO:tasks.rados.rados.0.smithi071.stdout:777: oid 48 version 167 is already newer than 167 2023-12-15T22:59:48.158 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 167 (ObjNum 373 snap 91 seq_num 373) dirty exists 2023-12-15T22:59:48.158 INFO:tasks.rados.rados.0.smithi071.stdout:777: left oid 48 (ObjNum 373 snap 91 seq_num 373) 2023-12-15T22:59:48.158 INFO:tasks.rados.rados.0.smithi071.stdout:785: finishing write tid 3 to smithi07183189-31 2023-12-15T22:59:48.158 INFO:tasks.rados.rados.0.smithi071.stdout:785: finishing write tid 4 to smithi07183189-31 2023-12-15T22:59:48.158 INFO:tasks.rados.rados.0.smithi071.stdout:785: oid 31 updating version 0 to 182 2023-12-15T22:59:48.158 INFO:tasks.rados.rados.0.smithi071.stdout:785: oid 31 updating version 182 to 184 2023-12-15T22:59:48.158 INFO:tasks.rados.rados.0.smithi071.stdout:785: oid 31 version 184 is already newer than 183 2023-12-15T22:59:48.158 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 31 v 184 (ObjNum 380 snap 91 seq_num 380) dirty exists 2023-12-15T22:59:48.158 INFO:tasks.rados.rados.0.smithi071.stdout:785: left oid 31 (ObjNum 380 snap 91 seq_num 380) 2023-12-15T22:59:48.342 INFO:tasks.rados.rados.0.smithi071.stdout:768: done (9 left) 2023-12-15T22:59:48.342 INFO:tasks.rados.rados.0.smithi071.stdout:770: done (8 left) 2023-12-15T22:59:48.342 INFO:tasks.rados.rados.0.smithi071.stdout:771: done (7 left) 2023-12-15T22:59:48.342 INFO:tasks.rados.rados.0.smithi071.stdout:774: done (6 left) 2023-12-15T22:59:48.342 INFO:tasks.rados.rados.0.smithi071.stdout:777: done (5 left) 2023-12-15T22:59:48.342 INFO:tasks.rados.rados.0.smithi071.stdout:779: done (4 left) 2023-12-15T22:59:48.342 INFO:tasks.rados.rados.0.smithi071.stdout:784: done (3 left) 2023-12-15T22:59:48.342 INFO:tasks.rados.rados.0.smithi071.stdout:785: done (2 left) 2023-12-15T22:59:48.342 INFO:tasks.rados.rados.0.smithi071.stdout:786: done (1 left) 2023-12-15T22:59:48.342 INFO:tasks.rados.rados.0.smithi071.stdout:787: done (0 left) 2023-12-15T22:59:48.342 INFO:tasks.rados.rados.0.smithi071.stdout:788: snap_create 2023-12-15T22:59:48.598 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:59:48.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:48.703+0000 7f5376020640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:48.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:48.707+0000 7f5376020640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:49.346 INFO:tasks.rados.rados.0.smithi071.stdout:788: done (0 left) 2023-12-15T22:59:49.346 INFO:tasks.rados.rados.0.smithi071.stdout:789: snap_remove snap 91 2023-12-15T22:59:49.571 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T22:59:49.571 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T22:59:49.571 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T22:59:49.680 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:49.676+0000 7fca338de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:49.683 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:49.683+0000 7fca338de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:49.861 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T22:59:49.861 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-12-15T22:59:49.872 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":295,"stamp":"2023-12-15T22:59:49.344497+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81348696,"num_objects":259,"num_object_clones":190,"num_object_copies":777,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":42,"num_read":1546,"num_read_kb":1095073,"num_write":1302,"num_write_kb":343883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2839,"ondisk_log_size":2839,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":306272,"kb_used_data":178176,"kb_used_omap":0,"kb_used_meta":128064,"kb_avail":377181088,"statfs":{"total":386547056640,"available":386233434112,"internally_reserved":0,"allocated":182452224,"data_stored":186355422,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":131137536},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4941324,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":16,"num_read_kb":15957,"num_write":15,"num_write_kb":5388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.008683"},"pg_stats":[{"pgid":"3.f","version":"185'90","reported_seq":458,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-12-15T22:59:43.373351+0000","last_change":"2023-12-15T22:59:41.394826+0000","last_active":"2023-12-15T22:59:43.373351+0000","last_peered":"2023-12-15T22:59:43.373351+0000","last_clean":"2023-12-15T22:59:43.373351+0000","last_became_active":"2023-12-15T22:56:39.503797+0000","last_became_peered":"2023-12-15T22:56:39.503797+0000","last_unstale":"2023-12-15T22:59:43.373351+0000","last_undegraded":"2023-12-15T22:59:43.373351+0000","last_fullsized":"2023-12-15T22:59:43.373351+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6991778,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":70,"num_read_kb":53991,"num_write":36,"num_write_kb":10667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":373,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-12-15T22:59:44.351883+0000","last_change":"2023-12-15T22:59:42.365549+0000","last_active":"2023-12-15T22:59:44.351883+0000","last_peered":"2023-12-15T22:59:44.351883+0000","last_clean":"2023-12-15T22:59:44.351883+0000","last_became_active":"2023-12-15T22:56:39.552856+0000","last_became_peered":"2023-12-15T22:56:39.552856+0000","last_unstale":"2023-12-15T22:59:44.351883+0000","last_undegraded":"2023-12-15T22:59:44.351883+0000","last_fullsized":"2023-12-15T22:59:44.351883+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"196'267","reported_seq":586,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-12-15T22:59:44.023211+0000","last_change":"2023-12-15T22:59:42.357418+0000","last_active":"2023-12-15T22:59:44.023211+0000","last_peered":"2023-12-15T22:59:44.023211+0000","last_clean":"2023-12-15T22:59:44.023211+0000","last_became_active":"2023-12-15T22:56:39.495490+0000","last_became_peered":"2023-12-15T22:56:39.495490+0000","last_unstale":"2023-12-15T22:59:44.023211+0000","last_undegraded":"2023-12-15T22:59:44.023211+0000","last_fullsized":"2023-12-15T22:59:44.023211+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":267,"log_dups_size":0,"ondisk_log_size":267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9496021,"num_objects":28,"num_object_clones":19,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":206,"num_read_kb":190164,"num_write":136,"num_write_kb":39043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"196'267","reported_seq":578,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-12-15T22:59:44.029421+0000","last_change":"2023-12-15T22:59:41.406252+0000","last_active":"2023-12-15T22:59:44.029421+0000","last_peered":"2023-12-15T22:59:44.029421+0000","last_clean":"2023-12-15T22:59:44.029421+0000","last_became_active":"2023-12-15T22:56:39.579234+0000","last_became_peered":"2023-12-15T22:56:39.579234+0000","last_unstale":"2023-12-15T22:59:44.029421+0000","last_undegraded":"2023-12-15T22:59:44.029421+0000","last_fullsized":"2023-12-15T22:59:44.029421+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":267,"log_dups_size":0,"ondisk_log_size":267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11781806,"num_objects":26,"num_object_clones":22,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":144,"num_read_kb":108443,"num_write":135,"num_write_kb":39553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"194'143","reported_seq":467,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-15T22:59:45.345570+0000","last_change":"2023-12-15T22:59:45.353559+0000","last_active":"2023-12-15T22:59:45.345570+0000","last_peered":"2023-12-15T22:59:45.345570+0000","last_clean":"2023-12-15T22:59:45.345570+0000","last_became_active":"2023-12-15T22:56:39.572176+0000","last_became_peered":"2023-12-15T22:56:39.572176+0000","last_unstale":"2023-12-15T22:59:45.345570+0000","last_undegraded":"2023-12-15T22:59:45.345570+0000","last_fullsized":"2023-12-15T22:59:45.345570+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7462374,"num_objects":13,"num_object_clones":9,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":57,"num_read_kb":29047,"num_write":60,"num_write_kb":18313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"192'139","reported_seq":462,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-12-15T22:59:43.379763+0000","last_change":"2023-12-15T22:59:41.395625+0000","last_active":"2023-12-15T22:59:43.379763+0000","last_peered":"2023-12-15T22:59:43.379763+0000","last_clean":"2023-12-15T22:59:43.379763+0000","last_became_active":"2023-12-15T22:56:39.516361+0000","last_became_peered":"2023-12-15T22:56:39.516361+0000","last_unstale":"2023-12-15T22:59:43.379763+0000","last_undegraded":"2023-12-15T22:59:43.379763+0000","last_fullsized":"2023-12-15T22:59:43.379763+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":51,"num_read_kb":38717,"num_write":51,"num_write_kb":14458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"194'539","reported_seq":677,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-12-15T22:59:43.396163+0000","last_change":"2023-12-15T22:59:41.396867+0000","last_active":"2023-12-15T22:59:43.396163+0000","last_peered":"2023-12-15T22:59:43.396163+0000","last_clean":"2023-12-15T22:59:43.396163+0000","last_became_active":"2023-12-15T22:56:39.519748+0000","last_became_peered":"2023-12-15T22:56:39.519748+0000","last_unstale":"2023-12-15T22:59:43.396163+0000","last_undegraded":"2023-12-15T22:59:43.396163+0000","last_fullsized":"2023-12-15T22:59:43.396163+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":539,"log_dups_size":0,"ondisk_log_size":539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13731451,"num_objects":48,"num_object_clones":38,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":206,"num_read_kb":127291,"num_write":205,"num_write_kb":59782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"184'127","reported_seq":464,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-12-15T22:59:43.371958+0000","last_change":"2023-12-15T22:59:42.354754+0000","last_active":"2023-12-15T22:59:43.371958+0000","last_peered":"2023-12-15T22:59:43.371958+0000","last_clean":"2023-12-15T22:59:43.371958+0000","last_became_active":"2023-12-15T22:56:39.543179+0000","last_became_peered":"2023-12-15T22:56:39.543179+0000","last_unstale":"2023-12-15T22:59:43.371958+0000","last_undegraded":"2023-12-15T22:59:43.371958+0000","last_fullsized":"2023-12-15T22:59:43.371958+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5949678,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":94,"num_read_kb":87445,"num_write":62,"num_write_kb":16612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":428,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-15T22:59:45.346617+0000","last_change":"2023-12-15T22:56:32.607017+0000","last_active":"2023-12-15T22:59:45.346617+0000","last_peered":"2023-12-15T22:59:45.346617+0000","last_clean":"2023-12-15T22:59:45.346617+0000","last_became_active":"2023-12-15T22:56:32.602384+0000","last_became_peered":"2023-12-15T22:56:32.602384+0000","last_unstale":"2023-12-15T22:59:45.346617+0000","last_undegraded":"2023-12-15T22:59:45.346617+0000","last_fullsized":"2023-12-15T22:59:45.346617+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_clean_scrub_stamp":"2023-12-15T22:56:31.525572+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":283,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-12-15T22:59:43.372909+0000","last_change":"2023-12-15T22:56:33.627481+0000","last_active":"2023-12-15T22:59:43.372909+0000","last_peered":"2023-12-15T22:59:43.372909+0000","last_clean":"2023-12-15T22:59:43.372909+0000","last_became_active":"2023-12-15T22:56:33.624689+0000","last_became_peered":"2023-12-15T22:56:33.624689+0000","last_unstale":"2023-12-15T22:59:43.372909+0000","last_undegraded":"2023-12-15T22:59:43.372909+0000","last_fullsized":"2023-12-15T22:59:43.372909+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"184'158","reported_seq":499,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-12-15T22:59:43.367855+0000","last_change":"2023-12-15T22:59:42.354444+0000","last_active":"2023-12-15T22:59:43.367855+0000","last_peered":"2023-12-15T22:59:43.367855+0000","last_clean":"2023-12-15T22:59:43.367855+0000","last_became_active":"2023-12-15T22:56:39.529359+0000","last_became_peered":"2023-12-15T22:56:39.529359+0000","last_unstale":"2023-12-15T22:59:43.367855+0000","last_undegraded":"2023-12-15T22:59:43.367855+0000","last_fullsized":"2023-12-15T22:59:43.367855+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9210269,"num_objects":15,"num_object_clones":9,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":117,"num_read_kb":86105,"num_write":84,"num_write_kb":25166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":311,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-12-15T22:59:43.367669+0000","last_change":"2023-12-15T22:56:33.655797+0000","last_active":"2023-12-15T22:59:43.367669+0000","last_peered":"2023-12-15T22:59:43.367669+0000","last_clean":"2023-12-15T22:59:43.367669+0000","last_became_active":"2023-12-15T22:56:33.652885+0000","last_became_peered":"2023-12-15T22:56:33.652885+0000","last_unstale":"2023-12-15T22:59:43.367669+0000","last_undegraded":"2023-12-15T22:59:43.367669+0000","last_fullsized":"2023-12-15T22:59:43.367669+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"198'244","reported_seq":534,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-15T22:59:45.349370+0000","last_change":"2023-12-15T22:59:45.371633+0000","last_active":"2023-12-15T22:59:45.349370+0000","last_peered":"2023-12-15T22:59:45.349370+0000","last_clean":"2023-12-15T22:59:45.349370+0000","last_became_active":"2023-12-15T22:56:39.543509+0000","last_became_peered":"2023-12-15T22:56:39.543509+0000","last_unstale":"2023-12-15T22:59:45.349370+0000","last_undegraded":"2023-12-15T22:59:45.349370+0000","last_fullsized":"2023-12-15T22:59:45.349370+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":244,"log_dups_size":0,"ondisk_log_size":244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2050487,"num_objects":24,"num_object_clones":17,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":137,"num_read_kb":105605,"num_write":90,"num_write_kb":27441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":283,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-12-15T22:59:43.354010+0000","last_change":"2023-12-15T22:56:33.600485+0000","last_active":"2023-12-15T22:59:43.354010+0000","last_peered":"2023-12-15T22:59:43.354010+0000","last_clean":"2023-12-15T22:59:43.354010+0000","last_became_active":"2023-12-15T22:56:33.597320+0000","last_became_peered":"2023-12-15T22:56:33.597320+0000","last_unstale":"2023-12-15T22:59:43.354010+0000","last_undegraded":"2023-12-15T22:59:43.354010+0000","last_fullsized":"2023-12-15T22:59:43.354010+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"198'115","reported_seq":450,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-15T22:59:45.345676+0000","last_change":"2023-12-15T22:59:45.371640+0000","last_active":"2023-12-15T22:59:45.345676+0000","last_peered":"2023-12-15T22:59:45.345676+0000","last_clean":"2023-12-15T22:59:45.345676+0000","last_became_active":"2023-12-15T22:56:39.560980+0000","last_became_peered":"2023-12-15T22:56:39.560980+0000","last_unstale":"2023-12-15T22:59:45.345676+0000","last_undegraded":"2023-12-15T22:59:45.345676+0000","last_fullsized":"2023-12-15T22:59:45.345676+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3815222,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":55,"num_read_kb":45341,"num_write":45,"num_write_kb":13643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":311,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-12-15T22:59:43.383312+0000","last_change":"2023-12-15T22:56:33.648541+0000","last_active":"2023-12-15T22:59:43.383312+0000","last_peered":"2023-12-15T22:59:43.383312+0000","last_clean":"2023-12-15T22:59:43.383312+0000","last_became_active":"2023-12-15T22:56:33.643681+0000","last_became_peered":"2023-12-15T22:56:33.643681+0000","last_unstale":"2023-12-15T22:59:43.383312+0000","last_undegraded":"2023-12-15T22:59:43.383312+0000","last_fullsized":"2023-12-15T22:59:43.383312+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"196'185","reported_seq":486,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-12-15T22:59:44.349628+0000","last_change":"2023-12-15T22:59:42.358136+0000","last_active":"2023-12-15T22:59:44.349628+0000","last_peered":"2023-12-15T22:59:44.349628+0000","last_clean":"2023-12-15T22:59:44.349628+0000","last_became_active":"2023-12-15T22:56:39.506510+0000","last_became_peered":"2023-12-15T22:56:39.506510+0000","last_unstale":"2023-12-15T22:59:44.349628+0000","last_undegraded":"2023-12-15T22:59:44.349628+0000","last_fullsized":"2023-12-15T22:59:44.349628+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2134522,"num_objects":18,"num_object_clones":13,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":100,"num_read_kb":71478,"num_write":71,"num_write_kb":20602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"185'98","reported_seq":407,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-12-15T22:59:43.377186+0000","last_change":"2023-12-15T22:59:42.357833+0000","last_active":"2023-12-15T22:59:43.377186+0000","last_peered":"2023-12-15T22:59:43.377186+0000","last_clean":"2023-12-15T22:59:43.377186+0000","last_became_active":"2023-12-15T22:56:39.511159+0000","last_became_peered":"2023-12-15T22:56:39.511159+0000","last_unstale":"2023-12-15T22:59:43.377186+0000","last_undegraded":"2023-12-15T22:59:43.377186+0000","last_fullsized":"2023-12-15T22:59:43.377186+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":21,"num_read_kb":7340,"num_write":37,"num_write_kb":11102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":289,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-12-15T22:59:44.348933+0000","last_change":"2023-12-15T22:56:33.633890+0000","last_active":"2023-12-15T22:59:44.348933+0000","last_peered":"2023-12-15T22:59:44.348933+0000","last_clean":"2023-12-15T22:59:44.348933+0000","last_became_active":"2023-12-15T22:56:33.631134+0000","last_became_peered":"2023-12-15T22:56:33.631134+0000","last_unstale":"2023-12-15T22:59:44.348933+0000","last_undegraded":"2023-12-15T22:59:44.348933+0000","last_fullsized":"2023-12-15T22:59:44.348933+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"196'126","reported_seq":447,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-12-15T22:59:44.348279+0000","last_change":"2023-12-15T22:59:42.366037+0000","last_active":"2023-12-15T22:59:44.348279+0000","last_peered":"2023-12-15T22:59:44.348279+0000","last_clean":"2023-12-15T22:59:44.348279+0000","last_became_active":"2023-12-15T22:56:39.532730+0000","last_became_peered":"2023-12-15T22:56:39.532730+0000","last_unstale":"2023-12-15T22:59:44.348279+0000","last_undegraded":"2023-12-15T22:59:44.348279+0000","last_fullsized":"2023-12-15T22:59:44.348279+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2016906,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":59,"num_read_kb":44847,"num_write":53,"num_write_kb":15885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":299,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-15T22:59:45.344246+0000","last_change":"2023-12-15T22:56:33.655736+0000","last_active":"2023-12-15T22:59:45.344246+0000","last_peered":"2023-12-15T22:59:45.344246+0000","last_clean":"2023-12-15T22:59:45.344246+0000","last_became_active":"2023-12-15T22:56:33.652917+0000","last_became_peered":"2023-12-15T22:56:33.652917+0000","last_unstale":"2023-12-15T22:59:45.344246+0000","last_undegraded":"2023-12-15T22:59:45.344246+0000","last_fullsized":"2023-12-15T22:59:45.344246+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"194'172","reported_seq":499,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-15T22:59:45.350044+0000","last_change":"2023-12-15T22:59:45.354729+0000","last_active":"2023-12-15T22:59:45.350044+0000","last_peered":"2023-12-15T22:59:45.350044+0000","last_clean":"2023-12-15T22:59:45.350044+0000","last_became_active":"2023-12-15T22:56:39.557252+0000","last_became_peered":"2023-12-15T22:56:39.557252+0000","last_unstale":"2023-12-15T22:59:45.350044+0000","last_undegraded":"2023-12-15T22:59:45.350044+0000","last_fullsized":"2023-12-15T22:59:45.350044+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2353013,"num_objects":14,"num_object_clones":10,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":95,"num_read_kb":69776,"num_write":73,"num_write_kb":21141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":289,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-12-15T22:59:44.350494+0000","last_change":"2023-12-15T22:56:33.632348+0000","last_active":"2023-12-15T22:59:44.350494+0000","last_peered":"2023-12-15T22:59:44.350494+0000","last_clean":"2023-12-15T22:59:44.350494+0000","last_became_active":"2023-12-15T22:56:33.629022+0000","last_became_peered":"2023-12-15T22:56:33.629022+0000","last_unstale":"2023-12-15T22:59:44.350494+0000","last_undegraded":"2023-12-15T22:59:44.350494+0000","last_fullsized":"2023-12-15T22:59:44.350494+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"187'72","reported_seq":412,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-12-15T22:59:44.344536+0000","last_change":"2023-12-15T22:59:42.366174+0000","last_active":"2023-12-15T22:59:44.344536+0000","last_peered":"2023-12-15T22:59:44.344536+0000","last_clean":"2023-12-15T22:59:44.344536+0000","last_became_active":"2023-12-15T22:56:39.535145+0000","last_became_peered":"2023-12-15T22:56:39.535145+0000","last_unstale":"2023-12-15T22:59:44.344536+0000","last_undegraded":"2023-12-15T22:59:44.344536+0000","last_fullsized":"2023-12-15T22:59:44.344536+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3764801,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":31,"num_read_kb":29392,"num_write":27,"num_write_kb":8581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":283,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-12-15T22:59:43.361762+0000","last_change":"2023-12-15T22:56:33.627005+0000","last_active":"2023-12-15T22:59:43.361762+0000","last_peered":"2023-12-15T22:59:43.361762+0000","last_clean":"2023-12-15T22:59:43.361762+0000","last_became_active":"2023-12-15T22:56:33.622181+0000","last_became_peered":"2023-12-15T22:56:33.622181+0000","last_unstale":"2023-12-15T22:59:43.361762+0000","last_undegraded":"2023-12-15T22:59:43.361762+0000","last_fullsized":"2023-12-15T22:59:43.361762+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":80758328,"num_objects":257,"num_object_clones":190,"num_object_copies":771,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":42,"num_read":1443,"num_read_kb":1094982,"num_write":1165,"num_write_kb":341989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2742,"ondisk_log_size":2742,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705859,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":84112,"kb_used_data":53128,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94287728,"statfs":{"total":96636764160,"available":96550633472,"internally_reserved":0,"allocated":54403072,"data_stored":56314909,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705858,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":68020,"kb_used_data":37036,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94303820,"statfs":{"total":96636764160,"available":96567111680,"internally_reserved":0,"allocated":37924864,"data_stored":38571854,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705858,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70148,"kb_used_data":39164,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94301692,"statfs":{"total":96636764160,"available":96564932608,"internally_reserved":0,"allocated":40103936,"data_stored":39443095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705859,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":83992,"kb_used_data":48848,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94287848,"statfs":{"total":96636764160,"available":96550756352,"internally_reserved":0,"allocated":50020352,"data_stored":52025564,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-15T22:59:50.347 INFO:tasks.rados.rados.0.smithi071.stdout:789: done (0 left) 2023-12-15T22:59:50.347 INFO:tasks.rados.rados.0.smithi071.stdout:790: write oid 9 current snap is 93 2023-12-15T22:59:50.347 INFO:tasks.rados.rados.0.smithi071.stdout:790: seq_num 381 ranges {403467=487519,1562110=579120} 2023-12-15T22:59:50.352 INFO:tasks.rados.rados.0.smithi071.stdout:790: writing smithi07183189-9 from 403467 to 890986 tid 1 2023-12-15T22:59:50.361 INFO:tasks.rados.rados.0.smithi071.stdout:790: writing smithi07183189-9 from 1562110 to 2141230 tid 2 2023-12-15T22:59:50.361 INFO:tasks.rados.rados.0.smithi071.stdout:791: read oid 33 snap -1 2023-12-15T22:59:50.361 INFO:tasks.rados.rados.0.smithi071.stdout:791: expect (ObjNum 367 snap 91 seq_num 367) 2023-12-15T22:59:50.361 INFO:tasks.rados.rados.0.smithi071.stdout:792: read oid 31 snap -1 2023-12-15T22:59:50.361 INFO:tasks.rados.rados.0.smithi071.stdout:792: expect (ObjNum 380 snap 91 seq_num 380) 2023-12-15T22:59:50.361 INFO:tasks.rados.rados.0.smithi071.stdout:793: write oid 29 current snap is 93 2023-12-15T22:59:50.361 INFO:tasks.rados.rados.0.smithi071.stdout:793: seq_num 382 ranges {420274=562768,1620592=439131} 2023-12-15T22:59:50.368 INFO:tasks.rados.rados.0.smithi071.stdout:793: writing smithi07183189-29 from 420274 to 983042 tid 1 2023-12-15T22:59:50.375 INFO:tasks.rados.rados.0.smithi071.stdout:793: writing smithi07183189-29 from 1620592 to 2059723 tid 2 2023-12-15T22:59:50.375 INFO:tasks.rados.rados.0.smithi071.stdout:794: delete oid 48 current snap is 93 2023-12-15T22:59:50.439 INFO:tasks.rados.rados.0.smithi071.stdout:794: done (4 left) 2023-12-15T22:59:50.439 INFO:tasks.rados.rados.0.smithi071.stdout:795: snap_remove snap 92 2023-12-15T22:59:50.775 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:59:50.878 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:50.877+0000 7f3e12e60640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:50.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:50.881+0000 7f3e12e60640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:51.358 INFO:tasks.rados.rados.0.smithi071.stdout:795: done (4 left) 2023-12-15T22:59:51.359 INFO:tasks.rados.rados.0.smithi071.stdout:796: read oid 21 snap -1 2023-12-15T22:59:51.359 INFO:tasks.rados.rados.0.smithi071.stdout:796: expect deleted 2023-12-15T22:59:51.359 INFO:tasks.rados.rados.0.smithi071.stdout:797: write oid 39 current snap is 93 2023-12-15T22:59:51.359 INFO:tasks.rados.rados.0.smithi071.stdout:797: seq_num 384 ranges {453888=713266,1737556=442800} 2023-12-15T22:59:51.367 INFO:tasks.rados.rados.0.smithi071.stdout:797: writing smithi07183189-39 from 453888 to 1167154 tid 1 2023-12-15T22:59:51.374 INFO:tasks.rados.rados.0.smithi071.stdout:797: writing smithi07183189-39 from 1737556 to 2180356 tid 2 2023-12-15T22:59:51.374 INFO:tasks.rados.rados.0.smithi071.stdout:798: write (excl) oid 44 current snap is 93 2023-12-15T22:59:51.374 INFO:tasks.rados.rados.0.smithi071.stdout:792: expect (ObjNum 380 snap 91 seq_num 380) 2023-12-15T22:59:51.509 INFO:tasks.rados.rados.0.smithi071.stdout:791: expect (ObjNum 367 snap 91 seq_num 367) 2023-12-15T22:59:51.629 INFO:tasks.rados.rados.0.smithi071.stdout:790: finishing write tid 1 to smithi07183189-9 2023-12-15T22:59:51.629 INFO:tasks.rados.rados.0.smithi071.stdout:798: seq_num 385 ranges {470695=788515,1679685=586458} 2023-12-15T22:59:51.635 INFO:tasks.rados.rados.0.smithi071.stdout:798: writing smithi07183189-44 from 470695 to 1259210 tid 1 2023-12-15T22:59:51.642 INFO:tasks.rados.rados.0.smithi071.stdout:798: writing smithi07183189-44 from 1679685 to 2266143 tid 2 2023-12-15T22:59:51.642 INFO:tasks.rados.rados.0.smithi071.stdout:791: done (6 left) 2023-12-15T22:59:51.642 INFO:tasks.rados.rados.0.smithi071.stdout:792: done (5 left) 2023-12-15T22:59:51.642 INFO:tasks.rados.rados.0.smithi071.stdout:799: delete oid 23 current snap is 93 2023-12-15T22:59:51.642 INFO:tasks.rados.rados.0.smithi071.stdout:790: finishing write tid 2 to smithi07183189-9 2023-12-15T22:59:51.642 INFO:tasks.rados.rados.0.smithi071.stdout:793: finishing write tid 1 to smithi07183189-29 2023-12-15T22:59:51.642 INFO:tasks.rados.rados.0.smithi071.stdout:790: finishing write tid 3 to smithi07183189-9 2023-12-15T22:59:51.643 INFO:tasks.rados.rados.0.smithi071.stdout:790: finishing write tid 4 to smithi07183189-9 2023-12-15T22:59:51.643 INFO:tasks.rados.rados.0.smithi071.stdout:790: oid 9 updating version 0 to 285 2023-12-15T22:59:51.643 INFO:tasks.rados.rados.0.smithi071.stdout:790: oid 9 updating version 285 to 286 2023-12-15T22:59:51.643 INFO:tasks.rados.rados.0.smithi071.stdout:790: oid 9 version 286 is already newer than 285 2023-12-15T22:59:51.643 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 286 (ObjNum 381 snap 93 seq_num 381) dirty exists 2023-12-15T22:59:51.643 INFO:tasks.rados.rados.0.smithi071.stdout:790: left oid 9 (ObjNum 381 snap 93 seq_num 381) 2023-12-15T22:59:51.643 INFO:tasks.rados.rados.0.smithi071.stdout:793: finishing write tid 2 to smithi07183189-29 2023-12-15T22:59:51.643 INFO:tasks.rados.rados.0.smithi071.stdout:793: finishing write tid 3 to smithi07183189-29 2023-12-15T22:59:51.643 INFO:tasks.rados.rados.0.smithi071.stdout:793: finishing write tid 4 to smithi07183189-29 2023-12-15T22:59:51.643 INFO:tasks.rados.rados.0.smithi071.stdout:793: oid 29 updating version 0 to 549 2023-12-15T22:59:51.643 INFO:tasks.rados.rados.0.smithi071.stdout:793: oid 29 version 549 is already newer than 548 2023-12-15T22:59:51.643 INFO:tasks.rados.rados.0.smithi071.stdout:793: oid 29 updating version 549 to 550 2023-12-15T22:59:51.643 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 550 (ObjNum 382 snap 93 seq_num 382) dirty exists 2023-12-15T22:59:51.643 INFO:tasks.rados.rados.0.smithi071.stdout:793: left oid 29 (ObjNum 382 snap 93 seq_num 382) 2023-12-15T22:59:51.643 INFO:tasks.rados.rados.0.smithi071.stdout:797: finishing write tid 1 to smithi07183189-39 2023-12-15T22:59:51.643 INFO:tasks.rados.rados.0.smithi071.stdout:797: finishing write tid 2 to smithi07183189-39 2023-12-15T22:59:51.644 INFO:tasks.rados.rados.0.smithi071.stdout:797: finishing write tid 3 to smithi07183189-39 2023-12-15T22:59:51.644 INFO:tasks.rados.rados.0.smithi071.stdout:797: finishing write tid 4 to smithi07183189-39 2023-12-15T22:59:51.644 INFO:tasks.rados.rados.0.smithi071.stdout:797: oid 39 updating version 0 to 191 2023-12-15T22:59:51.644 INFO:tasks.rados.rados.0.smithi071.stdout:797: oid 39 updating version 191 to 193 2023-12-15T22:59:51.644 INFO:tasks.rados.rados.0.smithi071.stdout:797: oid 39 version 193 is already newer than 193 2023-12-15T22:59:51.644 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 39 v 193 (ObjNum 384 snap 93 seq_num 384) dirty exists 2023-12-15T22:59:51.644 INFO:tasks.rados.rados.0.smithi071.stdout:797: left oid 39 (ObjNum 384 snap 93 seq_num 384) 2023-12-15T22:59:51.657 INFO:tasks.rados.rados.0.smithi071.stdout:790: done (5 left) 2023-12-15T22:59:51.657 INFO:tasks.rados.rados.0.smithi071.stdout:793: done (4 left) 2023-12-15T22:59:51.657 INFO:tasks.rados.rados.0.smithi071.stdout:796: done (3 left) 2023-12-15T22:59:51.657 INFO:tasks.rados.rados.0.smithi071.stdout:797: done (2 left) 2023-12-15T22:59:51.657 INFO:tasks.rados.rados.0.smithi071.stdout:799: done (1 left) 2023-12-15T22:59:51.657 INFO:tasks.rados.rados.0.smithi071.stdout:800: read oid 25 snap -1 2023-12-15T22:59:51.657 INFO:tasks.rados.rados.0.smithi071.stdout:800: expect (ObjNum 357 snap 90 seq_num 357) 2023-12-15T22:59:51.657 INFO:tasks.rados.rados.0.smithi071.stdout:801: read oid 42 snap -1 2023-12-15T22:59:51.658 INFO:tasks.rados.rados.0.smithi071.stdout:801: expect deleted 2023-12-15T22:59:51.658 INFO:tasks.rados.rados.0.smithi071.stdout:802: snap_create 2023-12-15T22:59:51.660 INFO:tasks.rados.rados.0.smithi071.stdout:798: finishing write tid 1 to smithi07183189-44 2023-12-15T22:59:51.673 INFO:tasks.rados.rados.0.smithi071.stdout:798: finishing write tid 2 to smithi07183189-44 2023-12-15T22:59:51.677 INFO:tasks.rados.rados.0.smithi071.stdout:800: expect (ObjNum 357 snap 90 seq_num 357) 2023-12-15T22:59:51.812 INFO:tasks.rados.rados.0.smithi071.stdout:798: finishing write tid 3 to smithi07183189-44 2023-12-15T22:59:51.812 INFO:tasks.rados.rados.0.smithi071.stdout:798: finishing write tid 4 to smithi07183189-44 2023-12-15T22:59:51.812 INFO:tasks.rados.rados.0.smithi071.stdout:798: oid 44 updating version 0 to 131 2023-12-15T22:59:51.812 INFO:tasks.rados.rados.0.smithi071.stdout:798: oid 44 updating version 131 to 133 2023-12-15T22:59:51.812 INFO:tasks.rados.rados.0.smithi071.stdout:798: oid 44 version 133 is already newer than 132 2023-12-15T22:59:51.812 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 133 (ObjNum 385 snap 93 seq_num 385) dirty exists 2023-12-15T22:59:51.812 INFO:tasks.rados.rados.0.smithi071.stdout:798: left oid 44 (ObjNum 385 snap 93 seq_num 385) 2023-12-15T22:59:52.357 INFO:tasks.rados.rados.0.smithi071.stdout:798: done (3 left) 2023-12-15T22:59:52.357 INFO:tasks.rados.rados.0.smithi071.stdout:800: done (2 left) 2023-12-15T22:59:52.357 INFO:tasks.rados.rados.0.smithi071.stdout:801: done (1 left) 2023-12-15T22:59:52.357 INFO:tasks.rados.rados.0.smithi071.stdout:802: done (0 left) 2023-12-15T22:59:52.357 INFO:tasks.rados.rados.0.smithi071.stdout:803: snap_remove snap 93 2023-12-15T22:59:52.948 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:59:53.054 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:53.053+0000 7fb4a8b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:53.057 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:53.057+0000 7fb4a8b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:53.364 INFO:tasks.rados.rados.0.smithi071.stdout:803: done (0 left) 2023-12-15T22:59:53.364 INFO:tasks.rados.rados.0.smithi071.stdout:804: read oid 29 snap -1 2023-12-15T22:59:53.364 INFO:tasks.rados.rados.0.smithi071.stdout:804: expect (ObjNum 382 snap 93 seq_num 382) 2023-12-15T22:59:53.364 INFO:tasks.rados.rados.0.smithi071.stdout:805: read oid 3 snap -1 2023-12-15T22:59:53.364 INFO:tasks.rados.rados.0.smithi071.stdout:805: expect (ObjNum 366 snap 91 seq_num 366) 2023-12-15T22:59:53.364 INFO:tasks.rados.rados.0.smithi071.stdout:806: snap_create 2023-12-15T22:59:53.411 INFO:tasks.rados.rados.0.smithi071.stdout:805: expect (ObjNum 366 snap 91 seq_num 366) 2023-12-15T22:59:53.494 INFO:tasks.rados.rados.0.smithi071.stdout:804: expect (ObjNum 382 snap 93 seq_num 382) 2023-12-15T22:59:54.368 INFO:tasks.rados.rados.0.smithi071.stdout:804: done (2 left) 2023-12-15T22:59:54.368 INFO:tasks.rados.rados.0.smithi071.stdout:805: done (1 left) 2023-12-15T22:59:54.368 INFO:tasks.rados.rados.0.smithi071.stdout:806: done (0 left) 2023-12-15T22:59:54.368 INFO:tasks.rados.rados.0.smithi071.stdout:807: delete oid 11 current snap is 95 2023-12-15T22:59:54.426 INFO:tasks.rados.rados.0.smithi071.stdout:807: done (0 left) 2023-12-15T22:59:54.427 INFO:tasks.rados.rados.0.smithi071.stdout:808: snap_remove snap 94 2023-12-15T22:59:54.875 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T22:59:54.875 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T22:59:54.875 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-12-15T22:59:54.875 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-12-15T22:59:54.994 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:54.989+0000 7febc4f04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:54.997 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:54.997+0000 7febc4f04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:55.010 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:59:55.010 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:59:55.010 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:59:55.010 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:59:55.010 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:59:55.010 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:59:55.010 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:59:55.010 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:59:55.010 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:59:55.010 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:59:55.010 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:59:55.010 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:59:55.012 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:59:55.013 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:59:55.013 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-12-15T22:59:55.119 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:55.114+0000 7f72e6155640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:55.122 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:55.122+0000 7f72e6155640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:55.127 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:59:55.135 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:59:55.135 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:59:55.135 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:59:55.135 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:59:55.135 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:59:55.135 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:59:55.135 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:59:55.135 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:59:55.135 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:59:55.135 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:59:55.135 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:59:55.135 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:59:55.138 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:59:55.138 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:59:55.138 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-12-15T22:59:55.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:55.231+0000 7fb04511e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:55.235 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:55.236+0000 7fb04511e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:55.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:55.291+0000 7f4100a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:55.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:55.298+0000 7f4100a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:55.320 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:59:55.320 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:59:55.320 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:59:55.321 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:59:55.321 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:59:55.321 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:59:55.321 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:59:55.321 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:59:55.321 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:59:55.321 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:59:55.321 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:59:55.321 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:59:55.323 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:59:55.323 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:59:55.323 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-12-15T22:59:55.379 INFO:tasks.rados.rados.0.smithi071.stdout:808: done (0 left) 2023-12-15T22:59:55.379 INFO:tasks.rados.rados.0.smithi071.stdout:809: read oid 44 snap -1 2023-12-15T22:59:55.380 INFO:tasks.rados.rados.0.smithi071.stdout:809: expect (ObjNum 385 snap 93 seq_num 385) 2023-12-15T22:59:55.380 INFO:tasks.rados.rados.0.smithi071.stdout:810: write oid 16 current snap is 95 2023-12-15T22:59:55.380 INFO:tasks.rados.rados.0.smithi071.stdout:810: seq_num 388 ranges {521116=730615,1971484=549632} 2023-12-15T22:59:55.388 INFO:tasks.rados.rados.0.smithi071.stdout:810: writing smithi07183189-16 from 521116 to 1251731 tid 1 2023-12-15T22:59:55.397 INFO:tasks.rados.rados.0.smithi071.stdout:810: writing smithi07183189-16 from 1971484 to 2521116 tid 2 2023-12-15T22:59:55.397 INFO:tasks.rados.rados.0.smithi071.stdout:811: read oid 24 snap -1 2023-12-15T22:59:55.397 INFO:tasks.rados.rados.0.smithi071.stdout:811: expect deleted 2023-12-15T22:59:55.397 INFO:tasks.rados.rados.0.smithi071.stdout:812: write oid 19 current snap is 95 2023-12-15T22:59:55.397 INFO:tasks.rados.rados.0.smithi071.stdout:812: seq_num 389 ranges {537923=405864,1513613=593796} 2023-12-15T22:59:55.403 INFO:tasks.rados.rados.0.smithi071.stdout:812: writing smithi07183189-19 from 537923 to 943787 tid 1 2023-12-15T22:59:55.411 INFO:tasks.rados.rados.0.smithi071.stdout:812: writing smithi07183189-19 from 1513613 to 2107409 tid 2 2023-12-15T22:59:55.411 INFO:tasks.rados.rados.0.smithi071.stdout:813: read oid 42 snap -1 2023-12-15T22:59:55.411 INFO:tasks.rados.rados.0.smithi071.stdout:813: expect deleted 2023-12-15T22:59:55.411 INFO:tasks.rados.rados.0.smithi071.stdout:814: write (excl) oid 11 current snap is 95 2023-12-15T22:59:55.411 INFO:tasks.rados.rados.0.smithi071.stdout:814: seq_num 390 ranges {554730=481113,1572095=737454} 2023-12-15T22:59:55.417 INFO:tasks.rados.rados.0.smithi071.stdout:814: writing smithi07183189-11 from 554730 to 1035843 tid 1 2023-12-15T22:59:55.425 INFO:tasks.rados.rados.0.smithi071.stdout:814: writing smithi07183189-11 from 1572095 to 2309549 tid 2 2023-12-15T22:59:55.425 INFO:tasks.rados.rados.0.smithi071.stdout:815: delete oid 13 current snap is 95 2023-12-15T22:59:55.440 INFO:tasks.rados.rados.0.smithi071.stdout:809: expect (ObjNum 385 snap 93 seq_num 385) 2023-12-15T22:59:55.484 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:55.479+0000 7fad5cb9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:55.488 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:55.488+0000 7fad5cb9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:55.507 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T22:59:55.507 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T22:59:55.507 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T22:59:55.507 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T22:59:55.508 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T22:59:55.508 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T22:59:55.508 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T22:59:55.508 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T22:59:55.508 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T22:59:55.508 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T22:59:55.508 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T22:59:55.508 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T22:59:55.510 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T22:59:55.510 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T22:59:55.592 INFO:tasks.rados.rados.0.smithi071.stdout:810: finishing write tid 1 to smithi07183189-16 2023-12-15T22:59:55.592 INFO:tasks.rados.rados.0.smithi071.stdout:812: finishing write tid 1 to smithi07183189-19 2023-12-15T22:59:55.592 INFO:tasks.rados.rados.0.smithi071.stdout:814: finishing write tid 1 to smithi07183189-11 2023-12-15T22:59:55.592 INFO:tasks.rados.rados.0.smithi071.stdout:810: finishing write tid 2 to smithi07183189-16 2023-12-15T22:59:55.592 INFO:tasks.rados.rados.0.smithi071.stdout:812: finishing write tid 2 to smithi07183189-19 2023-12-15T22:59:55.592 INFO:tasks.rados.rados.0.smithi071.stdout:814: finishing write tid 2 to smithi07183189-11 2023-12-15T22:59:55.592 INFO:tasks.rados.rados.0.smithi071.stdout:809: done (6 left) 2023-12-15T22:59:55.592 INFO:tasks.rados.rados.0.smithi071.stdout:811: done (5 left) 2023-12-15T22:59:55.592 INFO:tasks.rados.rados.0.smithi071.stdout:813: done (4 left) 2023-12-15T22:59:55.593 INFO:tasks.rados.rados.0.smithi071.stdout:815: done (3 left) 2023-12-15T22:59:55.593 INFO:tasks.rados.rados.0.smithi071.stdout:816: delete oid 14 current snap is 95 2023-12-15T22:59:55.593 INFO:tasks.rados.rados.0.smithi071.stdout:810: finishing write tid 3 to smithi07183189-16 2023-12-15T22:59:55.593 INFO:tasks.rados.rados.0.smithi071.stdout:810: finishing write tid 4 to smithi07183189-16 2023-12-15T22:59:55.593 INFO:tasks.rados.rados.0.smithi071.stdout:810: oid 16 updating version 0 to 279 2023-12-15T22:59:55.593 INFO:tasks.rados.rados.0.smithi071.stdout:810: oid 16 version 279 is already newer than 278 2023-12-15T22:59:55.593 INFO:tasks.rados.rados.0.smithi071.stdout:810: oid 16 updating version 279 to 280 2023-12-15T22:59:55.593 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 16 v 280 (ObjNum 388 snap 95 seq_num 388) dirty exists 2023-12-15T22:59:55.593 INFO:tasks.rados.rados.0.smithi071.stdout:810: left oid 16 (ObjNum 388 snap 95 seq_num 388) 2023-12-15T22:59:55.593 INFO:tasks.rados.rados.0.smithi071.stdout:814: finishing write tid 3 to smithi07183189-11 2023-12-15T22:59:55.593 INFO:tasks.rados.rados.0.smithi071.stdout:812: finishing write tid 3 to smithi07183189-19 2023-12-15T22:59:55.593 INFO:tasks.rados.rados.0.smithi071.stdout:814: finishing write tid 4 to smithi07183189-11 2023-12-15T22:59:55.593 INFO:tasks.rados.rados.0.smithi071.stdout:814: oid 11 updating version 0 to 177 2023-12-15T22:59:55.593 INFO:tasks.rados.rados.0.smithi071.stdout:814: oid 11 version 177 is already newer than 176 2023-12-15T22:59:55.594 INFO:tasks.rados.rados.0.smithi071.stdout:814: oid 11 updating version 177 to 178 2023-12-15T22:59:55.594 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 11 v 178 (ObjNum 390 snap 95 seq_num 390) dirty exists 2023-12-15T22:59:55.594 INFO:tasks.rados.rados.0.smithi071.stdout:814: left oid 11 (ObjNum 390 snap 95 seq_num 390) 2023-12-15T22:59:55.594 INFO:tasks.rados.rados.0.smithi071.stdout:812: finishing write tid 4 to smithi07183189-19 2023-12-15T22:59:55.594 INFO:tasks.rados.rados.0.smithi071.stdout:812: oid 19 updating version 0 to 290 2023-12-15T22:59:55.594 INFO:tasks.rados.rados.0.smithi071.stdout:812: oid 19 updating version 290 to 291 2023-12-15T22:59:55.594 INFO:tasks.rados.rados.0.smithi071.stdout:812: oid 19 version 291 is already newer than 289 2023-12-15T22:59:55.594 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 291 (ObjNum 389 snap 95 seq_num 389) dirty exists 2023-12-15T22:59:55.594 INFO:tasks.rados.rados.0.smithi071.stdout:812: left oid 19 (ObjNum 389 snap 95 seq_num 389) 2023-12-15T22:59:55.596 INFO:tasks.rados.rados.0.smithi071.stdout:810: done (3 left) 2023-12-15T22:59:55.596 INFO:tasks.rados.rados.0.smithi071.stdout:812: done (2 left) 2023-12-15T22:59:55.596 INFO:tasks.rados.rados.0.smithi071.stdout:814: done (1 left) 2023-12-15T22:59:55.597 INFO:tasks.rados.rados.0.smithi071.stdout:816: done (0 left) 2023-12-15T22:59:55.597 INFO:tasks.rados.rados.0.smithi071.stdout:817: read oid 45 snap -1 2023-12-15T22:59:55.597 INFO:tasks.rados.rados.0.smithi071.stdout:817: expect (ObjNum 315 snap 81 seq_num 315) 2023-12-15T22:59:55.597 INFO:tasks.rados.rados.0.smithi071.stdout:818: delete oid 25 current snap is 95 2023-12-15T22:59:55.614 INFO:tasks.rados.rados.0.smithi071.stdout:818: done (1 left) 2023-12-15T22:59:55.614 INFO:tasks.rados.rados.0.smithi071.stdout:819: read oid 15 snap -1 2023-12-15T22:59:55.614 INFO:tasks.rados.rados.0.smithi071.stdout:819: expect (ObjNum 369 snap 91 seq_num 369) 2023-12-15T22:59:55.614 INFO:tasks.rados.rados.0.smithi071.stdout:820: write (excl) oid 22 current snap is 95 2023-12-15T22:59:55.614 INFO:tasks.rados.rados.0.smithi071.stdout:820: seq_num 394 ranges {621958=782109,2089670=532288} 2023-12-15T22:59:55.624 INFO:tasks.rados.rados.0.smithi071.stdout:820: writing smithi07183189-22 from 621958 to 1404067 tid 1 2023-12-15T22:59:55.633 INFO:tasks.rados.rados.0.smithi071.stdout:820: writing smithi07183189-22 from 2089670 to 2621958 tid 2 2023-12-15T22:59:55.633 INFO:tasks.rados.rados.0.smithi071.stdout:821: snap_create 2023-12-15T22:59:55.634 INFO:tasks.rados.rados.0.smithi071.stdout:819: expect (ObjNum 369 snap 91 seq_num 369) 2023-12-15T22:59:55.748 INFO:tasks.rados.rados.0.smithi071.stdout:817: expect (ObjNum 315 snap 81 seq_num 315) 2023-12-15T22:59:55.879 INFO:tasks.rados.rados.0.smithi071.stdout:820: finishing write tid 1 to smithi07183189-22 2023-12-15T22:59:55.879 INFO:tasks.rados.rados.0.smithi071.stdout:820: finishing write tid 2 to smithi07183189-22 2023-12-15T22:59:55.880 INFO:tasks.rados.rados.0.smithi071.stdout:820: finishing write tid 3 to smithi07183189-22 2023-12-15T22:59:55.880 INFO:tasks.rados.rados.0.smithi071.stdout:820: finishing write tid 4 to smithi07183189-22 2023-12-15T22:59:55.880 INFO:tasks.rados.rados.0.smithi071.stdout:820: oid 22 updating version 0 to 196 2023-12-15T22:59:55.880 INFO:tasks.rados.rados.0.smithi071.stdout:820: oid 22 version 196 is already newer than 194 2023-12-15T22:59:55.880 INFO:tasks.rados.rados.0.smithi071.stdout:820: oid 22 version 196 is already newer than 195 2023-12-15T22:59:55.880 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 196 (ObjNum 394 snap 95 seq_num 394) dirty exists 2023-12-15T22:59:55.880 INFO:tasks.rados.rados.0.smithi071.stdout:820: left oid 22 (ObjNum 394 snap 95 seq_num 394) 2023-12-15T22:59:56.384 INFO:tasks.rados.rados.0.smithi071.stdout:817: done (3 left) 2023-12-15T22:59:56.384 INFO:tasks.rados.rados.0.smithi071.stdout:819: done (2 left) 2023-12-15T22:59:56.384 INFO:tasks.rados.rados.0.smithi071.stdout:820: done (1 left) 2023-12-15T22:59:56.384 INFO:tasks.rados.rados.0.smithi071.stdout:821: done (0 left) 2023-12-15T22:59:56.385 INFO:tasks.rados.rados.0.smithi071.stdout:822: delete oid 23 current snap is 96 2023-12-15T22:59:56.405 INFO:tasks.rados.rados.0.smithi071.stdout:822: done (0 left) 2023-12-15T22:59:56.406 INFO:tasks.rados.rados.0.smithi071.stdout:823: delete oid 27 current snap is 96 2023-12-15T22:59:56.432 INFO:tasks.rados.rados.0.smithi071.stdout:823: done (0 left) 2023-12-15T22:59:56.432 INFO:tasks.rados.rados.0.smithi071.stdout:824: delete oid 18 current snap is 96 2023-12-15T22:59:56.439 INFO:tasks.rados.rados.0.smithi071.stdout:824: done (0 left) 2023-12-15T22:59:56.439 INFO:tasks.rados.rados.0.smithi071.stdout:825: delete oid 39 current snap is 96 2023-12-15T22:59:56.455 INFO:tasks.rados.rados.0.smithi071.stdout:825: done (0 left) 2023-12-15T22:59:56.455 INFO:tasks.rados.rados.0.smithi071.stdout:826: snap_remove snap 95 2023-12-15T22:59:57.270 INFO:tasks.rados.rados.0.smithi071.stdout:826: done (0 left) 2023-12-15T22:59:57.271 INFO:tasks.rados.rados.0.smithi071.stdout:827: read oid 6 snap -1 2023-12-15T22:59:57.271 INFO:tasks.rados.rados.0.smithi071.stdout:827: expect (ObjNum 375 snap 91 seq_num 375) 2023-12-15T22:59:57.271 INFO:tasks.rados.rados.0.smithi071.stdout:828: delete oid 47 current snap is 96 2023-12-15T22:59:57.306 INFO:tasks.rados.rados.0.smithi071.stdout:828: done (1 left) 2023-12-15T22:59:57.306 INFO:tasks.rados.rados.0.smithi071.stdout:829: write oid 25 current snap is 96 2023-12-15T22:59:57.306 INFO:tasks.rados.rados.0.smithi071.stdout:829: seq_num 400 ranges {722800=549956,2040562=682238} 2023-12-15T22:59:57.314 INFO:tasks.rados.rados.0.smithi071.stdout:829: writing smithi07183189-25 from 722800 to 1272756 tid 1 2023-12-15T22:59:57.320 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T22:59:57.324 INFO:tasks.rados.rados.0.smithi071.stdout:829: writing smithi07183189-25 from 2040562 to 2722800 tid 2 2023-12-15T22:59:57.324 INFO:tasks.rados.rados.0.smithi071.stdout:830: snap_create 2023-12-15T22:59:57.350 INFO:tasks.rados.rados.0.smithi071.stdout:827: expect (ObjNum 375 snap 91 seq_num 375) 2023-12-15T22:59:57.433 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:57.431+0000 7f83698ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:57.435 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:57.436+0000 7f83698ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:57.569 INFO:tasks.rados.rados.0.smithi071.stdout:829: finishing write tid 1 to smithi07183189-25 2023-12-15T22:59:57.569 INFO:tasks.rados.rados.0.smithi071.stdout:829: finishing write tid 2 to smithi07183189-25 2023-12-15T22:59:57.569 INFO:tasks.rados.rados.0.smithi071.stdout:829: finishing write tid 3 to smithi07183189-25 2023-12-15T22:59:57.569 INFO:tasks.rados.rados.0.smithi071.stdout:829: finishing write tid 4 to smithi07183189-25 2023-12-15T22:59:57.569 INFO:tasks.rados.rados.0.smithi071.stdout:829: oid 25 updating version 0 to 282 2023-12-15T22:59:57.569 INFO:tasks.rados.rados.0.smithi071.stdout:829: oid 25 updating version 282 to 283 2023-12-15T22:59:57.570 INFO:tasks.rados.rados.0.smithi071.stdout:829: oid 25 updating version 283 to 284 2023-12-15T22:59:57.570 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 25 v 284 (ObjNum 400 snap 96 seq_num 400) dirty exists 2023-12-15T22:59:57.570 INFO:tasks.rados.rados.0.smithi071.stdout:829: left oid 25 (ObjNum 400 snap 96 seq_num 400) 2023-12-15T22:59:58.278 INFO:tasks.rados.rados.0.smithi071.stdout:827: done (2 left) 2023-12-15T22:59:58.278 INFO:tasks.rados.rados.0.smithi071.stdout:829: done (1 left) 2023-12-15T22:59:58.278 INFO:tasks.rados.rados.0.smithi071.stdout:830: done (0 left) 2023-12-15T22:59:58.278 INFO:tasks.rados.rados.0.smithi071.stdout:831: read oid 3 snap 96 2023-12-15T22:59:58.278 INFO:tasks.rados.rados.0.smithi071.stdout:831: expect (ObjNum 366 snap 91 seq_num 366) 2023-12-15T22:59:58.278 INFO:tasks.rados.rados.0.smithi071.stdout:832: write oid 49 current snap is 97 2023-12-15T22:59:58.278 INFO:tasks.rados.rados.0.smithi071.stdout:832: seq_num 401 ranges {739607=625205,1982691=499457} 2023-12-15T22:59:58.286 INFO:tasks.rados.rados.0.smithi071.stdout:832: writing smithi07183189-49 from 739607 to 1364812 tid 1 2023-12-15T22:59:58.293 INFO:tasks.rados.rados.0.smithi071.stdout:832: writing smithi07183189-49 from 1982691 to 2482148 tid 2 2023-12-15T22:59:58.294 INFO:tasks.rados.rados.0.smithi071.stdout:833: read oid 17 snap -1 2023-12-15T22:59:58.294 INFO:tasks.rados.rados.0.smithi071.stdout:833: expect (ObjNum 220 snap 49 seq_num 220) 2023-12-15T22:59:58.294 INFO:tasks.rados.rados.0.smithi071.stdout:834: snap_create 2023-12-15T22:59:58.313 INFO:tasks.rados.rados.0.smithi071.stdout:831: expect (ObjNum 366 snap 91 seq_num 366) 2023-12-15T22:59:58.410 INFO:tasks.rados.rados.0.smithi071.stdout:833: expect (ObjNum 220 snap 49 seq_num 220) 2023-12-15T22:59:58.692 INFO:tasks.rados.rados.0.smithi071.stdout:832: finishing write tid 1 to smithi07183189-49 2023-12-15T22:59:58.692 INFO:tasks.rados.rados.0.smithi071.stdout:832: finishing write tid 2 to smithi07183189-49 2023-12-15T22:59:58.692 INFO:tasks.rados.rados.0.smithi071.stdout:832: finishing write tid 3 to smithi07183189-49 2023-12-15T22:59:58.692 INFO:tasks.rados.rados.0.smithi071.stdout:832: finishing write tid 4 to smithi07183189-49 2023-12-15T22:59:58.692 INFO:tasks.rados.rados.0.smithi071.stdout:832: oid 49 updating version 0 to 204 2023-12-15T22:59:58.692 INFO:tasks.rados.rados.0.smithi071.stdout:832: oid 49 updating version 204 to 205 2023-12-15T22:59:58.692 INFO:tasks.rados.rados.0.smithi071.stdout:832: oid 49 version 205 is already newer than 203 2023-12-15T22:59:58.692 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 49 v 205 (ObjNum 401 snap 97 seq_num 401) dirty exists 2023-12-15T22:59:58.693 INFO:tasks.rados.rados.0.smithi071.stdout:832: left oid 49 (ObjNum 401 snap 97 seq_num 401) 2023-12-15T22:59:59.278 INFO:tasks.rados.rados.0.smithi071.stdout:831: done (3 left) 2023-12-15T22:59:59.279 INFO:tasks.rados.rados.0.smithi071.stdout:832: done (2 left) 2023-12-15T22:59:59.279 INFO:tasks.rados.rados.0.smithi071.stdout:833: done (1 left) 2023-12-15T22:59:59.279 INFO:tasks.rados.rados.0.smithi071.stdout:834: done (0 left) 2023-12-15T22:59:59.279 INFO:tasks.rados.rados.0.smithi071.stdout:835: read oid 1 snap -1 2023-12-15T22:59:59.279 INFO:tasks.rados.rados.0.smithi071.stdout:835: expect (ObjNum 342 snap 84 seq_num 342) 2023-12-15T22:59:59.279 INFO:tasks.rados.rados.0.smithi071.stdout:836: read oid 43 snap -1 2023-12-15T22:59:59.279 INFO:tasks.rados.rados.0.smithi071.stdout:836: expect (ObjNum 346 snap 86 seq_num 346) 2023-12-15T22:59:59.279 INFO:tasks.rados.rados.0.smithi071.stdout:837: read oid 41 snap -1 2023-12-15T22:59:59.279 INFO:tasks.rados.rados.0.smithi071.stdout:837: expect (ObjNum 365 snap 91 seq_num 365) 2023-12-15T22:59:59.279 INFO:tasks.rados.rados.0.smithi071.stdout:838: write (excl) oid 25 current snap is 98 2023-12-15T22:59:59.279 INFO:tasks.rados.rados.0.smithi071.stdout:838: seq_num 402 ranges {756414=700454,2041173=715241} 2023-12-15T22:59:59.288 INFO:tasks.rados.rados.0.smithi071.stdout:838: writing smithi07183189-25 from 756414 to 1456868 tid 1 2023-12-15T22:59:59.298 INFO:tasks.rados.rados.0.smithi071.stdout:838: writing smithi07183189-25 from 2041173 to 2756414 tid 2 2023-12-15T22:59:59.298 INFO:tasks.rados.rados.0.smithi071.stdout:839: delete oid 13 current snap is 98 2023-12-15T22:59:59.316 INFO:tasks.rados.rados.0.smithi071.stdout:836: expect (ObjNum 346 snap 86 seq_num 346) 2023-12-15T22:59:59.504 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T22:59:59.616 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:59.614+0000 7fe8f939e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:59.619 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T22:59:59.619+0000 7fe8f939e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T22:59:59.665 INFO:tasks.rados.rados.0.smithi071.stdout:837: expect (ObjNum 365 snap 91 seq_num 365) 2023-12-15T22:59:59.818 INFO:tasks.rados.rados.0.smithi071.stdout:835: expect (ObjNum 342 snap 84 seq_num 342) 2023-12-15T22:59:59.966 INFO:tasks.rados.rados.0.smithi071.stdout:838: finishing write tid 1 to smithi07183189-25 2023-12-15T22:59:59.966 INFO:tasks.rados.rados.0.smithi071.stdout:838: finishing write tid 2 to smithi07183189-25 2023-12-15T22:59:59.966 INFO:tasks.rados.rados.0.smithi071.stdout:838: finishing write tid 3 to smithi07183189-25 2023-12-15T22:59:59.966 INFO:tasks.rados.rados.0.smithi071.stdout:838: finishing write tid 4 to smithi07183189-25 2023-12-15T22:59:59.966 INFO:tasks.rados.rados.0.smithi071.stdout:838: oid 25 updating version 0 to 287 2023-12-15T22:59:59.966 INFO:tasks.rados.rados.0.smithi071.stdout:838: oid 25 version 287 is already newer than 286 2023-12-15T22:59:59.967 INFO:tasks.rados.rados.0.smithi071.stdout:838: oid 25 updating version 287 to 288 2023-12-15T22:59:59.967 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 25 v 288 (ObjNum 402 snap 98 seq_num 402) dirty exists 2023-12-15T22:59:59.967 INFO:tasks.rados.rados.0.smithi071.stdout:838: left oid 25 (ObjNum 402 snap 98 seq_num 402) 2023-12-15T22:59:59.967 INFO:tasks.rados.rados.0.smithi071.stdout:835: done (4 left) 2023-12-15T22:59:59.967 INFO:tasks.rados.rados.0.smithi071.stdout:836: done (3 left) 2023-12-15T22:59:59.967 INFO:tasks.rados.rados.0.smithi071.stdout:837: done (2 left) 2023-12-15T22:59:59.967 INFO:tasks.rados.rados.0.smithi071.stdout:838: done (1 left) 2023-12-15T22:59:59.967 INFO:tasks.rados.rados.0.smithi071.stdout:839: done (0 left) 2023-12-15T22:59:59.967 INFO:tasks.rados.rados.0.smithi071.stdout:840: read oid 22 snap 96 2023-12-15T22:59:59.967 INFO:tasks.rados.rados.0.smithi071.stdout:840: expect (ObjNum 394 snap 95 seq_num 394) 2023-12-15T22:59:59.967 INFO:tasks.rados.rados.0.smithi071.stdout:841: read oid 6 snap -1 2023-12-15T22:59:59.967 INFO:tasks.rados.rados.0.smithi071.stdout:841: expect (ObjNum 375 snap 91 seq_num 375) 2023-12-15T22:59:59.967 INFO:tasks.rados.rados.0.smithi071.stdout:842: read oid 24 snap 97 2023-12-15T22:59:59.967 INFO:tasks.rados.rados.0.smithi071.stdout:842: expect deleted 2023-12-15T22:59:59.967 INFO:tasks.rados.rados.0.smithi071.stdout:843: delete oid 28 current snap is 98 2023-12-15T22:59:59.985 INFO:tasks.rados.rados.0.smithi071.stdout:841: expect (ObjNum 375 snap 91 seq_num 375) 2023-12-15T23:00:00.192 INFO:tasks.rados.rados.0.smithi071.stdout:840: expect (ObjNum 394 snap 95 seq_num 394) 2023-12-15T23:00:00.294 INFO:tasks.rados.rados.0.smithi071.stdout:840: done (3 left) 2023-12-15T23:00:00.294 INFO:tasks.rados.rados.0.smithi071.stdout:841: done (2 left) 2023-12-15T23:00:00.295 INFO:tasks.rados.rados.0.smithi071.stdout:842: done (1 left) 2023-12-15T23:00:00.295 INFO:tasks.rados.rados.0.smithi071.stdout:843: done (0 left) 2023-12-15T23:00:00.295 INFO:tasks.rados.rados.0.smithi071.stdout:844: snap_remove snap 96 2023-12-15T23:00:00.511 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T23:00:00.511 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2023-12-15T23:00:00.620 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:00.615+0000 7f7dbe42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:00.623 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:00.623+0000 7f7dbe42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:01.278 INFO:teuthology.orchestra.run.smithi071.stderr:reweighted osd.0 to 1 (10000) 2023-12-15T23:00:01.281 INFO:tasks.rados.rados.0.smithi071.stdout:844: done (0 left) 2023-12-15T23:00:01.281 INFO:tasks.rados.rados.0.smithi071.stdout:845: write oid 45 current snap is 98 2023-12-15T23:00:01.282 INFO:tasks.rados.rados.0.smithi071.stdout:845: seq_num 405 ranges {406835=642554,1532972=506795,2772312=34523} 2023-12-15T23:00:01.289 INFO:tasks.rados.rados.0.smithi071.stdout:845: writing smithi07183189-45 from 406835 to 1049389 tid 1 2023-12-15T23:00:01.295 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2023-12-15T23:00:01.296 INFO:tasks.rados.rados.0.smithi071.stdout:845: writing smithi07183189-45 from 1532972 to 2039767 tid 2 2023-12-15T23:00:01.300 INFO:tasks.rados.rados.0.smithi071.stdout:845: writing smithi07183189-45 from 2772312 to 2806835 tid 3 2023-12-15T23:00:01.301 INFO:tasks.rados.rados.0.smithi071.stdout:846: write oid 46 current snap is 98 2023-12-15T23:00:01.301 INFO:tasks.rados.rados.0.smithi071.stdout:846: seq_num 406 ranges {423642=717803,1875101=766806} 2023-12-15T23:00:01.308 INFO:tasks.rados.rados.0.smithi071.stdout:846: writing smithi07183189-46 from 423642 to 1141445 tid 1 2023-12-15T23:00:01.318 INFO:tasks.rados.rados.0.smithi071.stdout:846: writing smithi07183189-46 from 1875101 to 2641907 tid 2 2023-12-15T23:00:01.318 INFO:tasks.rados.rados.0.smithi071.stdout:847: read oid 13 snap 97 2023-12-15T23:00:01.318 INFO:tasks.rados.rados.0.smithi071.stdout:847: expect deleted 2023-12-15T23:00:01.318 INFO:tasks.rados.rados.0.smithi071.stdout:848: write (excl) oid 1 current snap is 98 2023-12-15T23:00:01.318 INFO:tasks.rados.rados.0.smithi071.stdout:848: seq_num 407 ranges {440449=793052,1933583=510464} 2023-12-15T23:00:01.327 INFO:tasks.rados.rados.0.smithi071.stdout:848: writing smithi07183189-1 from 440449 to 1233501 tid 1 2023-12-15T23:00:01.337 INFO:tasks.rados.rados.0.smithi071.stdout:848: writing smithi07183189-1 from 1933583 to 2444047 tid 2 2023-12-15T23:00:01.337 INFO:tasks.rados.rados.0.smithi071.stdout:849: read oid 19 snap 97 2023-12-15T23:00:01.337 INFO:tasks.rados.rados.0.smithi071.stdout:849: expect (ObjNum 389 snap 95 seq_num 389) 2023-12-15T23:00:01.337 INFO:tasks.rados.rados.0.smithi071.stdout:850: write (excl) oid 42 current snap is 98 2023-12-15T23:00:01.337 INFO:tasks.rados.rados.0.smithi071.stdout:850: seq_num 408 ranges {457256=468301,1592065=770475,2854581=2675} 2023-12-15T23:00:01.340 INFO:tasks.rados.rados.0.smithi071.stdout:850: writing smithi07183189-42 from 457256 to 925557 tid 1 2023-12-15T23:00:01.350 INFO:tasks.rados.rados.0.smithi071.stdout:850: writing smithi07183189-42 from 1592065 to 2362540 tid 2 2023-12-15T23:00:01.353 INFO:tasks.rados.rados.0.smithi071.stdout:850: writing smithi07183189-42 from 2854581 to 2857256 tid 3 2023-12-15T23:00:01.353 INFO:tasks.rados.rados.0.smithi071.stdout:851: delete oid 29 current snap is 98 2023-12-15T23:00:01.411 INFO:tasks.rados.rados.0.smithi071.stdout:848: finishing write tid 1 to smithi07183189-1 2023-12-15T23:00:01.414 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:01.408+0000 7fe37b62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:01.415 INFO:tasks.rados.rados.0.smithi071.stdout:850: finishing write tid 1 to smithi07183189-42 2023-12-15T23:00:01.417 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:01.417+0000 7fe37b62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:01.435 INFO:tasks.rados.rados.0.smithi071.stdout:846: finishing write tid 1 to smithi07183189-46 2023-12-15T23:00:01.439 INFO:tasks.rados.rados.0.smithi071.stdout:845: finishing write tid 1 to smithi07183189-45 2023-12-15T23:00:01.468 INFO:tasks.rados.rados.0.smithi071.stdout:850: finishing write tid 2 to smithi07183189-42 2023-12-15T23:00:01.469 INFO:tasks.rados.rados.0.smithi071.stdout:849: expect (ObjNum 389 snap 95 seq_num 389) 2023-12-15T23:00:01.579 INFO:tasks.rados.rados.0.smithi071.stdout:848: finishing write tid 2 to smithi07183189-1 2023-12-15T23:00:01.579 INFO:tasks.rados.rados.0.smithi071.stdout:845: finishing write tid 2 to smithi07183189-45 2023-12-15T23:00:01.579 INFO:tasks.rados.rados.0.smithi071.stdout:847: done (6 left) 2023-12-15T23:00:01.579 INFO:tasks.rados.rados.0.smithi071.stdout:849: done (5 left) 2023-12-15T23:00:01.579 INFO:tasks.rados.rados.0.smithi071.stdout:851: done (4 left) 2023-12-15T23:00:01.579 INFO:tasks.rados.rados.0.smithi071.stdout:852: delete oid 10 current snap is 98 2023-12-15T23:00:01.579 INFO:tasks.rados.rados.0.smithi071.stdout:850: finishing write tid 3 to smithi07183189-42 2023-12-15T23:00:01.579 INFO:tasks.rados.rados.0.smithi071.stdout:846: finishing write tid 2 to smithi07183189-46 2023-12-15T23:00:01.579 INFO:tasks.rados.rados.0.smithi071.stdout:850: finishing write tid 4 to smithi07183189-42 2023-12-15T23:00:01.579 INFO:tasks.rados.rados.0.smithi071.stdout:848: finishing write tid 3 to smithi07183189-1 2023-12-15T23:00:01.579 INFO:tasks.rados.rados.0.smithi071.stdout:848: finishing write tid 4 to smithi07183189-1 2023-12-15T23:00:01.579 INFO:tasks.rados.rados.0.smithi071.stdout:848: oid 1 updating version 0 to 94 2023-12-15T23:00:01.579 INFO:tasks.rados.rados.0.smithi071.stdout:848: oid 1 version 94 is already newer than 92 2023-12-15T23:00:01.579 INFO:tasks.rados.rados.0.smithi071.stdout:848: oid 1 version 94 is already newer than 93 2023-12-15T23:00:01.580 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 1 v 94 (ObjNum 407 snap 98 seq_num 407) dirty exists 2023-12-15T23:00:01.580 INFO:tasks.rados.rados.0.smithi071.stdout:848: left oid 1 (ObjNum 407 snap 98 seq_num 407) 2023-12-15T23:00:01.580 INFO:tasks.rados.rados.0.smithi071.stdout:850: finishing write tid 5 to smithi07183189-42 2023-12-15T23:00:01.580 INFO:tasks.rados.rados.0.smithi071.stdout:850: oid 42 updating version 0 to 179 2023-12-15T23:00:01.580 INFO:tasks.rados.rados.0.smithi071.stdout:850: oid 42 updating version 179 to 182 2023-12-15T23:00:01.580 INFO:tasks.rados.rados.0.smithi071.stdout:850: oid 42 version 182 is already newer than 182 2023-12-15T23:00:01.580 INFO:tasks.rados.rados.0.smithi071.stdout:850: oid 42 version 182 is already newer than 180 2023-12-15T23:00:01.580 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 182 (ObjNum 408 snap 98 seq_num 408) dirty exists 2023-12-15T23:00:01.580 INFO:tasks.rados.rados.0.smithi071.stdout:850: left oid 42 (ObjNum 408 snap 98 seq_num 408) 2023-12-15T23:00:01.580 INFO:tasks.rados.rados.0.smithi071.stdout:845: finishing write tid 3 to smithi07183189-45 2023-12-15T23:00:01.580 INFO:tasks.rados.rados.0.smithi071.stdout:845: finishing write tid 4 to smithi07183189-45 2023-12-15T23:00:01.580 INFO:tasks.rados.rados.0.smithi071.stdout:846: finishing write tid 3 to smithi07183189-46 2023-12-15T23:00:01.580 INFO:tasks.rados.rados.0.smithi071.stdout:846: finishing write tid 4 to smithi07183189-46 2023-12-15T23:00:01.580 INFO:tasks.rados.rados.0.smithi071.stdout:846: oid 46 updating version 0 to 135 2023-12-15T23:00:01.580 INFO:tasks.rados.rados.0.smithi071.stdout:846: oid 46 version 135 is already newer than 134 2023-12-15T23:00:01.580 INFO:tasks.rados.rados.0.smithi071.stdout:846: oid 46 updating version 135 to 136 2023-12-15T23:00:01.580 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 136 (ObjNum 406 snap 98 seq_num 406) dirty exists 2023-12-15T23:00:01.581 INFO:tasks.rados.rados.0.smithi071.stdout:846: left oid 46 (ObjNum 406 snap 98 seq_num 406) 2023-12-15T23:00:01.581 INFO:tasks.rados.rados.0.smithi071.stdout:845: finishing write tid 5 to smithi07183189-45 2023-12-15T23:00:01.581 INFO:tasks.rados.rados.0.smithi071.stdout:845: oid 45 updating version 0 to 557 2023-12-15T23:00:01.581 INFO:tasks.rados.rados.0.smithi071.stdout:845: oid 45 version 557 is already newer than 555 2023-12-15T23:00:01.581 INFO:tasks.rados.rados.0.smithi071.stdout:845: oid 45 version 557 is already newer than 557 2023-12-15T23:00:01.581 INFO:tasks.rados.rados.0.smithi071.stdout:845: oid 45 version 557 is already newer than 554 2023-12-15T23:00:01.581 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 557 (ObjNum 405 snap 98 seq_num 405) dirty exists 2023-12-15T23:00:01.581 INFO:tasks.rados.rados.0.smithi071.stdout:845: left oid 45 (ObjNum 405 snap 98 seq_num 405) 2023-12-15T23:00:01.603 INFO:tasks.rados.rados.0.smithi071.stdout:845: done (4 left) 2023-12-15T23:00:01.603 INFO:tasks.rados.rados.0.smithi071.stdout:846: done (3 left) 2023-12-15T23:00:01.603 INFO:tasks.rados.rados.0.smithi071.stdout:848: done (2 left) 2023-12-15T23:00:01.603 INFO:tasks.rados.rados.0.smithi071.stdout:850: done (1 left) 2023-12-15T23:00:01.603 INFO:tasks.rados.rados.0.smithi071.stdout:852: done (0 left) 2023-12-15T23:00:01.603 INFO:tasks.rados.rados.0.smithi071.stdout:853: delete oid 31 current snap is 98 2023-12-15T23:00:01.630 INFO:tasks.rados.rados.0.smithi071.stdout:853: done (0 left) 2023-12-15T23:00:01.630 INFO:tasks.rados.rados.0.smithi071.stdout:854: read oid 22 snap -1 2023-12-15T23:00:01.630 INFO:tasks.rados.rados.0.smithi071.stdout:854: expect (ObjNum 394 snap 95 seq_num 394) 2023-12-15T23:00:01.630 INFO:tasks.rados.rados.0.smithi071.stdout:855: snap_remove snap 97 2023-12-15T23:00:01.713 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T23:00:01.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:01.815+0000 7fef4b59e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:01.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:01.820+0000 7fef4b59e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:02.267 INFO:teuthology.orchestra.run.smithi071.stderr:reweighted osd.1 to 1 (10000) 2023-12-15T23:00:02.270 INFO:tasks.rados.rados.0.smithi071.stdout:855: done (1 left) 2023-12-15T23:00:02.270 INFO:tasks.rados.rados.0.smithi071.stdout:856: delete oid 3 current snap is 98 2023-12-15T23:00:02.270 INFO:tasks.rados.rados.0.smithi071.stdout:854: expect (ObjNum 394 snap 95 seq_num 394) 2023-12-15T23:00:02.285 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2023-12-15T23:00:02.384 INFO:tasks.rados.rados.0.smithi071.stdout:854: done (1 left) 2023-12-15T23:00:02.384 INFO:tasks.rados.rados.0.smithi071.stdout:856: done (0 left) 2023-12-15T23:00:02.384 INFO:tasks.rados.rados.0.smithi071.stdout:857: delete oid 16 current snap is 98 2023-12-15T23:00:02.397 INFO:tasks.rados.rados.0.smithi071.stdout:857: done (0 left) 2023-12-15T23:00:02.397 INFO:tasks.rados.rados.0.smithi071.stdout:858: write oid 4 current snap is 98 2023-12-15T23:00:02.397 INFO:tasks.rados.rados.0.smithi071.stdout:858: seq_num 414 ranges {558098=636148,1826604=665129} 2023-12-15T23:00:02.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:02.394+0000 7ff01bd1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:02.403 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:02.403+0000 7ff01bd1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:02.403 INFO:tasks.rados.rados.0.smithi071.stdout:858: writing smithi07183189-4 from 558098 to 1194246 tid 1 2023-12-15T23:00:02.413 INFO:tasks.rados.rados.0.smithi071.stdout:858: writing smithi07183189-4 from 1826604 to 2491733 tid 2 2023-12-15T23:00:02.413 INFO:tasks.rados.rados.0.smithi071.stdout:859: read oid 11 snap -1 2023-12-15T23:00:02.413 INFO:tasks.rados.rados.0.smithi071.stdout:859: expect (ObjNum 390 snap 95 seq_num 390) 2023-12-15T23:00:02.413 INFO:tasks.rados.rados.0.smithi071.stdout:860: read oid 44 snap -1 2023-12-15T23:00:02.413 INFO:tasks.rados.rados.0.smithi071.stdout:860: expect (ObjNum 385 snap 93 seq_num 385) 2023-12-15T23:00:02.413 INFO:tasks.rados.rados.0.smithi071.stdout:861: delete oid 40 current snap is 98 2023-12-15T23:00:02.427 INFO:tasks.rados.rados.0.smithi071.stdout:860: expect (ObjNum 385 snap 93 seq_num 385) 2023-12-15T23:00:02.572 INFO:tasks.rados.rados.0.smithi071.stdout:859: expect (ObjNum 390 snap 95 seq_num 390) 2023-12-15T23:00:02.674 INFO:tasks.rados.rados.0.smithi071.stdout:858: finishing write tid 1 to smithi07183189-4 2023-12-15T23:00:02.674 INFO:tasks.rados.rados.0.smithi071.stdout:858: finishing write tid 2 to smithi07183189-4 2023-12-15T23:00:02.675 INFO:tasks.rados.rados.0.smithi071.stdout:858: finishing write tid 3 to smithi07183189-4 2023-12-15T23:00:02.675 INFO:tasks.rados.rados.0.smithi071.stdout:858: finishing write tid 4 to smithi07183189-4 2023-12-15T23:00:02.675 INFO:tasks.rados.rados.0.smithi071.stdout:858: oid 4 updating version 0 to 296 2023-12-15T23:00:02.675 INFO:tasks.rados.rados.0.smithi071.stdout:858: oid 4 version 296 is already newer than 295 2023-12-15T23:00:02.675 INFO:tasks.rados.rados.0.smithi071.stdout:858: oid 4 version 296 is already newer than 294 2023-12-15T23:00:02.675 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 4 v 296 (ObjNum 414 snap 98 seq_num 414) dirty exists 2023-12-15T23:00:02.675 INFO:tasks.rados.rados.0.smithi071.stdout:858: left oid 4 (ObjNum 414 snap 98 seq_num 414) 2023-12-15T23:00:02.675 INFO:tasks.rados.rados.0.smithi071.stdout:858: done (3 left) 2023-12-15T23:00:02.675 INFO:tasks.rados.rados.0.smithi071.stdout:859: done (2 left) 2023-12-15T23:00:02.675 INFO:tasks.rados.rados.0.smithi071.stdout:860: done (1 left) 2023-12-15T23:00:02.675 INFO:tasks.rados.rados.0.smithi071.stdout:861: done (0 left) 2023-12-15T23:00:02.675 INFO:tasks.rados.rados.0.smithi071.stdout:862: read oid 29 snap -1 2023-12-15T23:00:02.675 INFO:tasks.rados.rados.0.smithi071.stdout:862: expect deleted 2023-12-15T23:00:02.675 INFO:tasks.rados.rados.0.smithi071.stdout:863: write (excl) oid 21 current snap is 98 2023-12-15T23:00:02.675 INFO:tasks.rados.rados.0.smithi071.stdout:863: seq_num 416 ranges {591712=786646,1943568=668798} 2023-12-15T23:00:02.682 INFO:tasks.rados.rados.0.smithi071.stdout:863: writing smithi07183189-21 from 591712 to 1378358 tid 1 2023-12-15T23:00:02.689 INFO:tasks.rados.rados.0.smithi071.stdout:863: writing smithi07183189-21 from 1943568 to 2612366 tid 2 2023-12-15T23:00:02.689 INFO:tasks.rados.rados.0.smithi071.stdout:864: read oid 26 snap -1 2023-12-15T23:00:02.690 INFO:tasks.rados.rados.0.smithi071.stdout:864: expect (ObjNum 361 snap 91 seq_num 361) 2023-12-15T23:00:02.690 INFO:tasks.rados.rados.0.smithi071.stdout:865: snap_create 2023-12-15T23:00:02.742 INFO:tasks.rados.rados.0.smithi071.stdout:863: finishing write tid 1 to smithi07183189-21 2023-12-15T23:00:02.743 INFO:tasks.rados.rados.0.smithi071.stdout:864: expect (ObjNum 361 snap 91 seq_num 361) 2023-12-15T23:00:02.869 INFO:tasks.rados.rados.0.smithi071.stdout:863: finishing write tid 2 to smithi07183189-21 2023-12-15T23:00:02.869 INFO:tasks.rados.rados.0.smithi071.stdout:863: finishing write tid 3 to smithi07183189-21 2023-12-15T23:00:02.869 INFO:tasks.rados.rados.0.smithi071.stdout:863: finishing write tid 4 to smithi07183189-21 2023-12-15T23:00:02.870 INFO:tasks.rados.rados.0.smithi071.stdout:863: oid 21 updating version 0 to 256 2023-12-15T23:00:02.870 INFO:tasks.rados.rados.0.smithi071.stdout:863: oid 21 updating version 256 to 258 2023-12-15T23:00:02.870 INFO:tasks.rados.rados.0.smithi071.stdout:863: oid 21 version 258 is already newer than 257 2023-12-15T23:00:02.870 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 258 (ObjNum 416 snap 98 seq_num 416) dirty exists 2023-12-15T23:00:02.870 INFO:tasks.rados.rados.0.smithi071.stdout:863: left oid 21 (ObjNum 416 snap 98 seq_num 416) 2023-12-15T23:00:03.286 INFO:teuthology.orchestra.run.smithi071.stderr:reweighted osd.2 to 1 (10000) 2023-12-15T23:00:03.293 INFO:tasks.rados.rados.0.smithi071.stdout:862: done (3 left) 2023-12-15T23:00:03.293 INFO:tasks.rados.rados.0.smithi071.stdout:863: done (2 left) 2023-12-15T23:00:03.293 INFO:tasks.rados.rados.0.smithi071.stdout:864: done (1 left) 2023-12-15T23:00:03.293 INFO:tasks.rados.rados.0.smithi071.stdout:865: done (0 left) 2023-12-15T23:00:03.293 INFO:tasks.rados.rados.0.smithi071.stdout:866: delete oid 17 current snap is 99 2023-12-15T23:00:03.300 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2023-12-15T23:00:03.348 INFO:tasks.rados.rados.0.smithi071.stdout:866: done (0 left) 2023-12-15T23:00:03.348 INFO:tasks.rados.rados.0.smithi071.stdout:867: read oid 49 snap -1 2023-12-15T23:00:03.348 INFO:tasks.rados.rados.0.smithi071.stdout:867: expect (ObjNum 401 snap 97 seq_num 401) 2023-12-15T23:00:03.348 INFO:tasks.rados.rados.0.smithi071.stdout:868: delete oid 28 current snap is 99 2023-12-15T23:00:03.373 INFO:tasks.rados.rados.0.smithi071.stdout:867: expect (ObjNum 401 snap 97 seq_num 401) 2023-12-15T23:00:03.411 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:03.406+0000 7f3753915640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:03.414 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:03.414+0000 7f3753915640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:03.549 INFO:tasks.rados.rados.0.smithi071.stdout:867: done (1 left) 2023-12-15T23:00:03.549 INFO:tasks.rados.rados.0.smithi071.stdout:868: done (0 left) 2023-12-15T23:00:03.549 INFO:tasks.rados.rados.0.smithi071.stdout:869: read oid 11 snap -1 2023-12-15T23:00:03.549 INFO:tasks.rados.rados.0.smithi071.stdout:869: expect (ObjNum 390 snap 95 seq_num 390) 2023-12-15T23:00:03.549 INFO:tasks.rados.rados.0.smithi071.stdout:870: write oid 48 current snap is 99 2023-12-15T23:00:03.549 INFO:tasks.rados.rados.0.smithi071.stdout:870: seq_num 419 ranges {642133=728746,2119014=532478} 2023-12-15T23:00:03.557 INFO:tasks.rados.rados.0.smithi071.stdout:870: writing smithi07183189-48 from 642133 to 1370879 tid 1 2023-12-15T23:00:03.566 INFO:tasks.rados.rados.0.smithi071.stdout:870: writing smithi07183189-48 from 2119014 to 2651492 tid 2 2023-12-15T23:00:03.566 INFO:tasks.rados.rados.0.smithi071.stdout:871: write (excl) oid 27 current snap is 99 2023-12-15T23:00:03.566 INFO:tasks.rados.rados.0.smithi071.stdout:871: seq_num 420 ranges {658940=403995,1777496=676136,2950951=107989} 2023-12-15T23:00:03.573 INFO:tasks.rados.rados.0.smithi071.stdout:871: writing smithi07183189-27 from 658940 to 1062935 tid 1 2023-12-15T23:00:03.583 INFO:tasks.rados.rados.0.smithi071.stdout:871: writing smithi07183189-27 from 1777496 to 2453632 tid 2 2023-12-15T23:00:03.587 INFO:tasks.rados.rados.0.smithi071.stdout:871: writing smithi07183189-27 from 2950951 to 3058940 tid 3 2023-12-15T23:00:03.587 INFO:tasks.rados.rados.0.smithi071.stdout:872: write oid 17 current snap is 99 2023-12-15T23:00:03.587 INFO:tasks.rados.rados.0.smithi071.stdout:872: seq_num 421 ranges {675747=479244,1835978=536147} 2023-12-15T23:00:03.594 INFO:tasks.rados.rados.0.smithi071.stdout:872: writing smithi07183189-17 from 675747 to 1154991 tid 1 2023-12-15T23:00:03.603 INFO:tasks.rados.rados.0.smithi071.stdout:872: writing smithi07183189-17 from 1835978 to 2372125 tid 2 2023-12-15T23:00:03.603 INFO:tasks.rados.rados.0.smithi071.stdout:873: write (excl) oid 4 current snap is 99 2023-12-15T23:00:03.603 INFO:tasks.rados.rados.0.smithi071.stdout:873: seq_num 422 ranges {692554=554493,1778107=679805,2889444=203110} 2023-12-15T23:00:03.611 INFO:tasks.rados.rados.0.smithi071.stdout:873: writing smithi07183189-4 from 692554 to 1247047 tid 1 2023-12-15T23:00:03.620 INFO:tasks.rados.rados.0.smithi071.stdout:873: writing smithi07183189-4 from 1778107 to 2457912 tid 2 2023-12-15T23:00:03.625 INFO:tasks.rados.rados.0.smithi071.stdout:873: writing smithi07183189-4 from 2889444 to 3092554 tid 3 2023-12-15T23:00:03.625 INFO:tasks.rados.rados.0.smithi071.stdout:874: write (excl) oid 47 current snap is 99 2023-12-15T23:00:03.625 INFO:tasks.rados.rados.0.smithi071.stdout:874: seq_num 423 ranges {709361=629742,1836589=539816,3033220=76141} 2023-12-15T23:00:03.634 INFO:tasks.rados.rados.0.smithi071.stdout:874: writing smithi07183189-47 from 709361 to 1339103 tid 1 2023-12-15T23:00:03.641 INFO:tasks.rados.rados.0.smithi071.stdout:874: writing smithi07183189-47 from 1836589 to 2376405 tid 2 2023-12-15T23:00:03.646 INFO:tasks.rados.rados.0.smithi071.stdout:874: writing smithi07183189-47 from 3033220 to 3109361 tid 3 2023-12-15T23:00:03.646 INFO:tasks.rados.rados.0.smithi071.stdout:875: write oid 3 current snap is 99 2023-12-15T23:00:03.646 INFO:tasks.rados.rados.0.smithi071.stdout:869: expect (ObjNum 390 snap 95 seq_num 390) 2023-12-15T23:00:03.762 INFO:tasks.rados.rados.0.smithi071.stdout:871: finishing write tid 1 to smithi07183189-27 2023-12-15T23:00:03.762 INFO:tasks.rados.rados.0.smithi071.stdout:871: finishing write tid 2 to smithi07183189-27 2023-12-15T23:00:03.762 INFO:tasks.rados.rados.0.smithi071.stdout:870: finishing write tid 1 to smithi07183189-48 2023-12-15T23:00:03.762 INFO:tasks.rados.rados.0.smithi071.stdout:871: finishing write tid 3 to smithi07183189-27 2023-12-15T23:00:03.762 INFO:tasks.rados.rados.0.smithi071.stdout:872: finishing write tid 1 to smithi07183189-17 2023-12-15T23:00:03.762 INFO:tasks.rados.rados.0.smithi071.stdout:871: finishing write tid 4 to smithi07183189-27 2023-12-15T23:00:03.762 INFO:tasks.rados.rados.0.smithi071.stdout:871: finishing write tid 5 to smithi07183189-27 2023-12-15T23:00:03.762 INFO:tasks.rados.rados.0.smithi071.stdout:871: oid 27 updating version 0 to 210 2023-12-15T23:00:03.762 INFO:tasks.rados.rados.0.smithi071.stdout:871: oid 27 version 210 is already newer than 208 2023-12-15T23:00:03.762 INFO:tasks.rados.rados.0.smithi071.stdout:871: oid 27 updating version 210 to 211 2023-12-15T23:00:03.762 INFO:tasks.rados.rados.0.smithi071.stdout:871: oid 27 version 211 is already newer than 210 2023-12-15T23:00:03.762 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 211 (ObjNum 420 snap 99 seq_num 420) dirty exists 2023-12-15T23:00:03.762 INFO:tasks.rados.rados.0.smithi071.stdout:871: left oid 27 (ObjNum 420 snap 99 seq_num 420) 2023-12-15T23:00:03.762 INFO:tasks.rados.rados.0.smithi071.stdout:874: finishing write tid 1 to smithi07183189-47 2023-12-15T23:00:03.762 INFO:tasks.rados.rados.0.smithi071.stdout:870: finishing write tid 2 to smithi07183189-48 2023-12-15T23:00:03.763 INFO:tasks.rados.rados.0.smithi071.stdout:872: finishing write tid 2 to smithi07183189-17 2023-12-15T23:00:03.763 INFO:tasks.rados.rados.0.smithi071.stdout:870: finishing write tid 3 to smithi07183189-48 2023-12-15T23:00:03.763 INFO:tasks.rados.rados.0.smithi071.stdout:874: finishing write tid 2 to smithi07183189-47 2023-12-15T23:00:03.763 INFO:tasks.rados.rados.0.smithi071.stdout:870: finishing write tid 4 to smithi07183189-48 2023-12-15T23:00:03.763 INFO:tasks.rados.rados.0.smithi071.stdout:870: oid 48 updating version 0 to 187 2023-12-15T23:00:03.763 INFO:tasks.rados.rados.0.smithi071.stdout:870: oid 48 updating version 187 to 188 2023-12-15T23:00:03.763 INFO:tasks.rados.rados.0.smithi071.stdout:870: oid 48 updating version 188 to 189 2023-12-15T23:00:03.763 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 189 (ObjNum 419 snap 99 seq_num 419) dirty exists 2023-12-15T23:00:03.763 INFO:tasks.rados.rados.0.smithi071.stdout:870: left oid 48 (ObjNum 419 snap 99 seq_num 419) 2023-12-15T23:00:03.763 INFO:tasks.rados.rados.0.smithi071.stdout:872: finishing write tid 3 to smithi07183189-17 2023-12-15T23:00:03.763 INFO:tasks.rados.rados.0.smithi071.stdout:872: finishing write tid 4 to smithi07183189-17 2023-12-15T23:00:03.763 INFO:tasks.rados.rados.0.smithi071.stdout:872: oid 17 updating version 0 to 301 2023-12-15T23:00:03.763 INFO:tasks.rados.rados.0.smithi071.stdout:872: oid 17 version 301 is already newer than 300 2023-12-15T23:00:03.763 INFO:tasks.rados.rados.0.smithi071.stdout:872: oid 17 version 301 is already newer than 299 2023-12-15T23:00:03.763 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 17 v 301 (ObjNum 421 snap 99 seq_num 421) dirty exists 2023-12-15T23:00:03.764 INFO:tasks.rados.rados.0.smithi071.stdout:872: left oid 17 (ObjNum 421 snap 99 seq_num 421) 2023-12-15T23:00:03.764 INFO:tasks.rados.rados.0.smithi071.stdout:874: finishing write tid 3 to smithi07183189-47 2023-12-15T23:00:03.764 INFO:tasks.rados.rados.0.smithi071.stdout:875: seq_num 424 ranges {726168=704991,1895071=683474} 2023-12-15T23:00:03.769 INFO:tasks.rados.rados.0.smithi071.stdout:875: writing smithi07183189-3 from 726168 to 1431159 tid 1 2023-12-15T23:00:03.778 INFO:tasks.rados.rados.0.smithi071.stdout:875: writing smithi07183189-3 from 1895071 to 2578545 tid 2 2023-12-15T23:00:03.778 INFO:tasks.rados.rados.0.smithi071.stdout:869: done (6 left) 2023-12-15T23:00:03.778 INFO:tasks.rados.rados.0.smithi071.stdout:870: done (5 left) 2023-12-15T23:00:03.778 INFO:tasks.rados.rados.0.smithi071.stdout:871: done (4 left) 2023-12-15T23:00:03.778 INFO:tasks.rados.rados.0.smithi071.stdout:872: done (3 left) 2023-12-15T23:00:03.778 INFO:tasks.rados.rados.0.smithi071.stdout:873: finishing write tid 1 to smithi07183189-4 2023-12-15T23:00:03.778 INFO:tasks.rados.rados.0.smithi071.stdout:874: finishing write tid 4 to smithi07183189-47 2023-12-15T23:00:03.778 INFO:tasks.rados.rados.0.smithi071.stdout:874: finishing write tid 5 to smithi07183189-47 2023-12-15T23:00:03.778 INFO:tasks.rados.rados.0.smithi071.stdout:874: oid 47 updating version 0 to 154 2023-12-15T23:00:03.778 INFO:tasks.rados.rados.0.smithi071.stdout:874: oid 47 version 154 is already newer than 153 2023-12-15T23:00:03.779 INFO:tasks.rados.rados.0.smithi071.stdout:874: oid 47 updating version 154 to 155 2023-12-15T23:00:03.779 INFO:tasks.rados.rados.0.smithi071.stdout:874: oid 47 version 155 is already newer than 152 2023-12-15T23:00:03.779 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 155 (ObjNum 423 snap 99 seq_num 423) dirty exists 2023-12-15T23:00:03.779 INFO:tasks.rados.rados.0.smithi071.stdout:874: left oid 47 (ObjNum 423 snap 99 seq_num 423) 2023-12-15T23:00:03.779 INFO:tasks.rados.rados.0.smithi071.stdout:873: finishing write tid 2 to smithi07183189-4 2023-12-15T23:00:03.779 INFO:tasks.rados.rados.0.smithi071.stdout:873: finishing write tid 3 to smithi07183189-4 2023-12-15T23:00:03.779 INFO:tasks.rados.rados.0.smithi071.stdout:873: finishing write tid 4 to smithi07183189-4 2023-12-15T23:00:03.779 INFO:tasks.rados.rados.0.smithi071.stdout:873: finishing write tid 5 to smithi07183189-4 2023-12-15T23:00:03.779 INFO:tasks.rados.rados.0.smithi071.stdout:873: oid 4 updating version 0 to 306 2023-12-15T23:00:03.779 INFO:tasks.rados.rados.0.smithi071.stdout:873: oid 4 version 306 is already newer than 303 2023-12-15T23:00:03.779 INFO:tasks.rados.rados.0.smithi071.stdout:873: oid 4 version 306 is already newer than 306 2023-12-15T23:00:03.779 INFO:tasks.rados.rados.0.smithi071.stdout:873: oid 4 version 306 is already newer than 306 2023-12-15T23:00:03.779 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 4 v 306 (ObjNum 422 snap 99 seq_num 422) dirty exists 2023-12-15T23:00:03.779 INFO:tasks.rados.rados.0.smithi071.stdout:873: left oid 4 (ObjNum 422 snap 99 seq_num 422) 2023-12-15T23:00:03.779 INFO:tasks.rados.rados.0.smithi071.stdout:876: read oid 18 snap -1 2023-12-15T23:00:03.779 INFO:tasks.rados.rados.0.smithi071.stdout:876: expect deleted 2023-12-15T23:00:03.779 INFO:tasks.rados.rados.0.smithi071.stdout:873: done (3 left) 2023-12-15T23:00:03.780 INFO:tasks.rados.rados.0.smithi071.stdout:874: done (2 left) 2023-12-15T23:00:03.780 INFO:tasks.rados.rados.0.smithi071.stdout:877: write (excl) oid 45 current snap is 99 2023-12-15T23:00:03.780 INFO:tasks.rados.rados.0.smithi071.stdout:877: seq_num 425 ranges {742975=780240,1953553=427132,2971713=171262} 2023-12-15T23:00:03.788 INFO:tasks.rados.rados.0.smithi071.stdout:877: writing smithi07183189-45 from 742975 to 1523215 tid 1 2023-12-15T23:00:03.794 INFO:tasks.rados.rados.0.smithi071.stdout:877: writing smithi07183189-45 from 1953553 to 2380685 tid 2 2023-12-15T23:00:03.799 INFO:tasks.rados.rados.0.smithi071.stdout:877: writing smithi07183189-45 from 2971713 to 3142975 tid 3 2023-12-15T23:00:03.799 INFO:tasks.rados.rados.0.smithi071.stdout:878: snap_create 2023-12-15T23:00:03.800 INFO:tasks.rados.rados.0.smithi071.stdout:875: finishing write tid 1 to smithi07183189-3 2023-12-15T23:00:03.815 INFO:tasks.rados.rados.0.smithi071.stdout:875: finishing write tid 2 to smithi07183189-3 2023-12-15T23:00:03.842 INFO:tasks.rados.rados.0.smithi071.stdout:875: finishing write tid 3 to smithi07183189-3 2023-12-15T23:00:03.842 INFO:tasks.rados.rados.0.smithi071.stdout:875: finishing write tid 4 to smithi07183189-3 2023-12-15T23:00:03.842 INFO:tasks.rados.rados.0.smithi071.stdout:875: oid 3 updating version 0 to 136 2023-12-15T23:00:03.842 INFO:tasks.rados.rados.0.smithi071.stdout:875: oid 3 updating version 136 to 137 2023-12-15T23:00:03.842 INFO:tasks.rados.rados.0.smithi071.stdout:875: oid 3 version 137 is already newer than 135 2023-12-15T23:00:03.842 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 137 (ObjNum 424 snap 99 seq_num 424) dirty exists 2023-12-15T23:00:03.842 INFO:tasks.rados.rados.0.smithi071.stdout:875: left oid 3 (ObjNum 424 snap 99 seq_num 424) 2023-12-15T23:00:03.842 INFO:tasks.rados.rados.0.smithi071.stdout:877: finishing write tid 1 to smithi07183189-45 2023-12-15T23:00:03.859 INFO:tasks.rados.rados.0.smithi071.stdout:877: finishing write tid 2 to smithi07183189-45 2023-12-15T23:00:03.862 INFO:tasks.rados.rados.0.smithi071.stdout:877: finishing write tid 3 to smithi07183189-45 2023-12-15T23:00:03.870 INFO:tasks.rados.rados.0.smithi071.stdout:877: finishing write tid 4 to smithi07183189-45 2023-12-15T23:00:03.871 INFO:tasks.rados.rados.0.smithi071.stdout:877: finishing write tid 5 to smithi07183189-45 2023-12-15T23:00:03.871 INFO:tasks.rados.rados.0.smithi071.stdout:877: oid 45 updating version 0 to 566 2023-12-15T23:00:03.871 INFO:tasks.rados.rados.0.smithi071.stdout:877: oid 45 updating version 566 to 568 2023-12-15T23:00:03.871 INFO:tasks.rados.rados.0.smithi071.stdout:877: oid 45 version 568 is already newer than 568 2023-12-15T23:00:03.871 INFO:tasks.rados.rados.0.smithi071.stdout:877: oid 45 updating version 568 to 569 2023-12-15T23:00:03.871 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 569 (ObjNum 425 snap 99 seq_num 425) dirty exists 2023-12-15T23:00:03.872 INFO:tasks.rados.rados.0.smithi071.stdout:877: left oid 45 (ObjNum 425 snap 99 seq_num 425) 2023-12-15T23:00:03.887 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T23:00:03.990 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:03.989+0000 7fda3c46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:03.993 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:03.993+0000 7fda3c46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:04.287 INFO:teuthology.orchestra.run.smithi071.stderr:reweighted osd.3 to 1 (10000) 2023-12-15T23:00:04.292 INFO:tasks.rados.rados.0.smithi071.stdout:875: done (3 left) 2023-12-15T23:00:04.292 INFO:tasks.rados.rados.0.smithi071.stdout:876: done (2 left) 2023-12-15T23:00:04.292 INFO:tasks.rados.rados.0.smithi071.stdout:877: done (1 left) 2023-12-15T23:00:04.292 INFO:tasks.rados.rados.0.smithi071.stdout:878: done (0 left) 2023-12-15T23:00:04.292 INFO:tasks.rados.rados.0.smithi071.stdout:879: write (excl) oid 48 current snap is 100 2023-12-15T23:00:04.292 INFO:tasks.rados.rados.0.smithi071.stdout:879: seq_num 426 ranges {759782=571842,2012035=687143} 2023-12-15T23:00:04.301 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2023-12-15T23:00:04.302 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T23:00:04.302 INFO:tasks.rados.rados.0.smithi071.stdout:879: writing smithi07183189-48 from 759782 to 1331624 tid 1 2023-12-15T23:00:04.311 INFO:tasks.rados.rados.0.smithi071.stdout:879: writing smithi07183189-48 from 2012035 to 2699178 tid 2 2023-12-15T23:00:04.311 INFO:tasks.rados.rados.0.smithi071.stdout:880: delete oid 24 current snap is 100 2023-12-15T23:00:04.340 INFO:tasks.rados.rados.0.smithi071.stdout:880: done (1 left) 2023-12-15T23:00:04.340 INFO:tasks.rados.rados.0.smithi071.stdout:881: snap_remove snap 99 2023-12-15T23:00:04.413 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:04.409+0000 7fe0b08ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:04.416 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:04.416+0000 7fe0b08ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:04.592 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T23:00:04.592 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-12-15T23:00:04.604 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":317,"stamp":"2023-12-15T23:00:04.553068+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92408692,"num_objects":281,"num_object_clones":200,"num_object_copies":843,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":46,"num_read":1693,"num_read_kb":1190731,"num_write":1410,"num_write_kb":371985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2982,"ondisk_log_size":2982,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":306672,"kb_used_data":174416,"kb_used_omap":0,"kb_used_meta":132224,"kb_avail":377180688,"statfs":{"total":386547056640,"available":386233024512,"internally_reserved":0,"allocated":178601984,"data_stored":179887777,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":135397376},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3563117,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":25,"num_read_kb":25028,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000717"},"pg_stats":[{"pgid":"3.f","version":"185'90","reported_seq":495,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.348271+0000","last_change":"2023-12-15T22:59:57.312208+0000","last_active":"2023-12-15T22:59:59.348271+0000","last_peered":"2023-12-15T22:59:59.348271+0000","last_clean":"2023-12-15T22:59:59.348271+0000","last_became_active":"2023-12-15T22:56:39.503797+0000","last_became_peered":"2023-12-15T22:56:39.503797+0000","last_unstale":"2023-12-15T22:59:59.348271+0000","last_undegraded":"2023-12-15T22:59:59.348271+0000","last_fullsized":"2023-12-15T22:59:59.348271+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6991778,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":76,"num_read_kb":60668,"num_write":36,"num_write_kb":10667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":403,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.547183+0000","last_change":"2023-12-15T22:59:58.326705+0000","last_active":"2023-12-15T22:59:59.547183+0000","last_peered":"2023-12-15T22:59:59.547183+0000","last_clean":"2023-12-15T22:59:59.547183+0000","last_became_active":"2023-12-15T22:56:39.552856+0000","last_became_peered":"2023-12-15T22:56:39.552856+0000","last_unstale":"2023-12-15T22:59:59.547183+0000","last_undegraded":"2023-12-15T22:59:59.547183+0000","last_fullsized":"2023-12-15T22:59:59.547183+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"212'288","reported_seq":647,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.400744+0000","last_change":"2023-12-15T22:59:57.338044+0000","last_active":"2023-12-15T22:59:59.400744+0000","last_peered":"2023-12-15T22:59:59.400744+0000","last_clean":"2023-12-15T22:59:59.400744+0000","last_became_active":"2023-12-15T22:56:39.495490+0000","last_became_peered":"2023-12-15T22:56:39.495490+0000","last_unstale":"2023-12-15T22:59:59.400744+0000","last_undegraded":"2023-12-15T22:59:59.400744+0000","last_fullsized":"2023-12-15T22:59:59.400744+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":288,"log_dups_size":0,"ondisk_log_size":288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17227439,"num_objects":32,"num_object_clones":20,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":226,"num_read_kb":202764,"num_write":156,"num_write_kb":44891,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"208'291","reported_seq":625,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.325227+0000","last_change":"2023-12-15T22:59:57.330188+0000","last_active":"2023-12-15T22:59:59.325227+0000","last_peered":"2023-12-15T22:59:59.325227+0000","last_clean":"2023-12-15T22:59:59.325227+0000","last_became_active":"2023-12-15T22:56:39.579234+0000","last_became_peered":"2023-12-15T22:56:39.579234+0000","last_unstale":"2023-12-15T22:59:59.325227+0000","last_undegraded":"2023-12-15T22:59:59.325227+0000","last_fullsized":"2023-12-15T22:59:59.325227+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8891068,"num_objects":28,"num_object_clones":24,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":149,"num_read_kb":108446,"num_write":147,"num_write_kb":42462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"210'144","reported_seq":499,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.287440+0000","last_change":"2023-12-15T22:59:57.296538+0000","last_active":"2023-12-15T22:59:59.287440+0000","last_peered":"2023-12-15T22:59:59.287440+0000","last_clean":"2023-12-15T22:59:59.287440+0000","last_became_active":"2023-12-15T22:56:39.572176+0000","last_became_peered":"2023-12-15T22:56:39.572176+0000","last_unstale":"2023-12-15T22:59:59.287440+0000","last_undegraded":"2023-12-15T22:59:59.287440+0000","last_fullsized":"2023-12-15T22:59:59.287440+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3899257,"num_objects":14,"num_object_clones":10,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":58,"num_read_kb":29047,"num_write":60,"num_write_kb":18313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"209'141","reported_seq":496,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.322006+0000","last_change":"2023-12-15T22:59:57.320770+0000","last_active":"2023-12-15T22:59:59.322006+0000","last_peered":"2023-12-15T22:59:59.322006+0000","last_clean":"2023-12-15T22:59:59.322006+0000","last_became_active":"2023-12-15T22:56:39.516361+0000","last_became_peered":"2023-12-15T22:56:39.516361+0000","last_unstale":"2023-12-15T22:59:59.322006+0000","last_undegraded":"2023-12-15T22:59:59.322006+0000","last_fullsized":"2023-12-15T22:59:59.322006+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":51,"num_read_kb":38717,"num_write":51,"num_write_kb":14458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"204'552","reported_seq":735,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.322641+0000","last_change":"2023-12-15T22:59:57.320651+0000","last_active":"2023-12-15T22:59:59.322641+0000","last_peered":"2023-12-15T22:59:59.322641+0000","last_clean":"2023-12-15T22:59:59.322641+0000","last_became_active":"2023-12-15T22:56:39.519748+0000","last_became_peered":"2023-12-15T22:56:39.519748+0000","last_unstale":"2023-12-15T22:59:59.322641+0000","last_undegraded":"2023-12-15T22:59:59.322641+0000","last_fullsized":"2023-12-15T22:59:59.322641+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":552,"log_dups_size":0,"ondisk_log_size":552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13933267,"num_objects":50,"num_object_clones":39,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":235,"num_read_kb":149617,"num_write":217,"num_write_kb":62654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"200'133","reported_seq":515,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.305402+0000","last_change":"2023-12-15T22:59:57.321958+0000","last_active":"2023-12-15T22:59:59.305402+0000","last_peered":"2023-12-15T22:59:59.305402+0000","last_clean":"2023-12-15T22:59:59.305402+0000","last_became_active":"2023-12-15T22:56:39.543179+0000","last_became_peered":"2023-12-15T22:56:39.543179+0000","last_unstale":"2023-12-15T22:59:59.305402+0000","last_undegraded":"2023-12-15T22:59:59.305402+0000","last_fullsized":"2023-12-15T22:59:59.305402+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7361565,"num_objects":13,"num_object_clones":9,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":116,"num_read_kb":103738,"num_write":70,"num_write_kb":18597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":452,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.294380+0000","last_change":"2023-12-15T22:56:32.607017+0000","last_active":"2023-12-15T22:59:59.294380+0000","last_peered":"2023-12-15T22:59:59.294380+0000","last_clean":"2023-12-15T22:59:59.294380+0000","last_became_active":"2023-12-15T22:56:32.602384+0000","last_became_peered":"2023-12-15T22:56:32.602384+0000","last_unstale":"2023-12-15T22:59:59.294380+0000","last_undegraded":"2023-12-15T22:59:59.294380+0000","last_fullsized":"2023-12-15T22:59:59.294380+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_clean_scrub_stamp":"2023-12-15T22:56:31.525572+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":309,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.303877+0000","last_change":"2023-12-15T22:56:33.627481+0000","last_active":"2023-12-15T22:59:59.303877+0000","last_peered":"2023-12-15T22:59:59.303877+0000","last_clean":"2023-12-15T22:59:59.303877+0000","last_became_active":"2023-12-15T22:56:33.624689+0000","last_became_peered":"2023-12-15T22:56:33.624689+0000","last_unstale":"2023-12-15T22:59:59.303877+0000","last_undegraded":"2023-12-15T22:59:59.303877+0000","last_fullsized":"2023-12-15T22:59:59.303877+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"208'178","reported_seq":557,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.309217+0000","last_change":"2023-12-15T22:59:57.304470+0000","last_active":"2023-12-15T22:59:59.309217+0000","last_peered":"2023-12-15T22:59:59.309217+0000","last_clean":"2023-12-15T22:59:59.309217+0000","last_became_active":"2023-12-15T22:56:39.529359+0000","last_became_peered":"2023-12-15T22:56:39.529359+0000","last_unstale":"2023-12-15T22:59:59.309217+0000","last_undegraded":"2023-12-15T22:59:59.309217+0000","last_fullsized":"2023-12-15T22:59:59.309217+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6941489,"num_objects":19,"num_object_clones":11,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":137,"num_read_kb":97172,"num_write":100,"num_write_kb":28930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":337,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.320394+0000","last_change":"2023-12-15T22:56:33.655797+0000","last_active":"2023-12-15T22:59:59.320394+0000","last_peered":"2023-12-15T22:59:59.320394+0000","last_clean":"2023-12-15T22:59:59.320394+0000","last_became_active":"2023-12-15T22:56:33.652885+0000","last_became_peered":"2023-12-15T22:56:33.652885+0000","last_unstale":"2023-12-15T22:59:59.320394+0000","last_undegraded":"2023-12-15T22:59:59.320394+0000","last_fullsized":"2023-12-15T22:59:59.320394+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"208'254","reported_seq":576,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.321298+0000","last_change":"2023-12-15T22:59:57.298807+0000","last_active":"2023-12-15T22:59:59.321298+0000","last_peered":"2023-12-15T22:59:59.321298+0000","last_clean":"2023-12-15T22:59:59.321298+0000","last_became_active":"2023-12-15T22:56:39.543509+0000","last_became_peered":"2023-12-15T22:56:39.543509+0000","last_unstale":"2023-12-15T22:59:59.321298+0000","last_undegraded":"2023-12-15T22:59:59.321298+0000","last_fullsized":"2023-12-15T22:59:59.321298+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":254,"log_dups_size":0,"ondisk_log_size":254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4386759,"num_objects":25,"num_object_clones":17,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":145,"num_read_kb":111254,"num_write":98,"num_write_kb":29596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":309,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.309106+0000","last_change":"2023-12-15T22:56:33.600485+0000","last_active":"2023-12-15T22:59:59.309106+0000","last_peered":"2023-12-15T22:59:59.309106+0000","last_clean":"2023-12-15T22:59:59.309106+0000","last_became_active":"2023-12-15T22:56:33.597320+0000","last_became_peered":"2023-12-15T22:56:33.597320+0000","last_unstale":"2023-12-15T22:59:59.309106+0000","last_undegraded":"2023-12-15T22:59:59.309106+0000","last_fullsized":"2023-12-15T22:59:59.309106+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"200'120","reported_seq":488,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.309031+0000","last_change":"2023-12-15T22:59:57.297889+0000","last_active":"2023-12-15T22:59:59.309031+0000","last_peered":"2023-12-15T22:59:59.309031+0000","last_clean":"2023-12-15T22:59:59.309031+0000","last_became_active":"2023-12-15T22:56:39.560980+0000","last_became_peered":"2023-12-15T22:56:39.560980+0000","last_unstale":"2023-12-15T22:59:59.309031+0000","last_undegraded":"2023-12-15T22:59:59.309031+0000","last_fullsized":"2023-12-15T22:59:59.309031+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6134654,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":62,"num_read_kb":52203,"num_write":49,"num_write_kb":14551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":337,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.323730+0000","last_change":"2023-12-15T22:56:33.648541+0000","last_active":"2023-12-15T22:59:59.323730+0000","last_peered":"2023-12-15T22:59:59.323730+0000","last_clean":"2023-12-15T22:59:59.323730+0000","last_became_active":"2023-12-15T22:56:33.643681+0000","last_became_peered":"2023-12-15T22:56:33.643681+0000","last_unstale":"2023-12-15T22:59:59.323730+0000","last_undegraded":"2023-12-15T22:59:59.323730+0000","last_fullsized":"2023-12-15T22:59:59.323730+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"211'205","reported_seq":532,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.541673+0000","last_change":"2023-12-15T22:59:58.387917+0000","last_active":"2023-12-15T22:59:59.541673+0000","last_peered":"2023-12-15T22:59:59.541673+0000","last_clean":"2023-12-15T22:59:59.541673+0000","last_became_active":"2023-12-15T22:56:39.506510+0000","last_became_peered":"2023-12-15T22:56:39.506510+0000","last_unstale":"2023-12-15T22:59:59.541673+0000","last_undegraded":"2023-12-15T22:59:59.541673+0000","last_fullsized":"2023-12-15T22:59:59.541673+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5361565,"num_objects":22,"num_object_clones":15,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":111,"num_read_kb":75801,"num_write":83,"num_write_kb":24115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"185'98","reported_seq":441,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.302204+0000","last_change":"2023-12-15T22:59:57.336227+0000","last_active":"2023-12-15T22:59:59.302204+0000","last_peered":"2023-12-15T22:59:59.302204+0000","last_clean":"2023-12-15T22:59:59.302204+0000","last_became_active":"2023-12-15T22:56:39.511159+0000","last_became_peered":"2023-12-15T22:56:39.511159+0000","last_unstale":"2023-12-15T22:59:59.302204+0000","last_undegraded":"2023-12-15T22:59:59.302204+0000","last_fullsized":"2023-12-15T22:59:59.302204+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":21,"num_read_kb":7340,"num_write":37,"num_write_kb":11102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":311,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.545607+0000","last_change":"2023-12-15T22:56:33.633890+0000","last_active":"2023-12-15T22:59:59.545607+0000","last_peered":"2023-12-15T22:59:59.545607+0000","last_clean":"2023-12-15T22:59:59.545607+0000","last_became_active":"2023-12-15T22:56:33.631134+0000","last_became_peered":"2023-12-15T22:56:33.631134+0000","last_unstale":"2023-12-15T22:59:59.545607+0000","last_undegraded":"2023-12-15T22:59:59.545607+0000","last_fullsized":"2023-12-15T22:59:59.545607+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"204'133","reported_seq":484,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.543977+0000","last_change":"2023-12-15T22:59:58.326883+0000","last_active":"2023-12-15T22:59:59.543977+0000","last_peered":"2023-12-15T22:59:59.543977+0000","last_clean":"2023-12-15T22:59:59.543977+0000","last_became_active":"2023-12-15T22:56:39.532730+0000","last_became_peered":"2023-12-15T22:56:39.532730+0000","last_unstale":"2023-12-15T22:59:59.543977+0000","last_undegraded":"2023-12-15T22:59:59.543977+0000","last_fullsized":"2023-12-15T22:59:59.543977+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2470695,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":67,"num_read_kb":49963,"num_write":57,"num_write_kb":17229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":323,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.293952+0000","last_change":"2023-12-15T22:56:33.655736+0000","last_active":"2023-12-15T22:59:59.293952+0000","last_peered":"2023-12-15T22:59:59.293952+0000","last_clean":"2023-12-15T22:59:59.293952+0000","last_became_active":"2023-12-15T22:56:33.652917+0000","last_became_peered":"2023-12-15T22:56:33.652917+0000","last_unstale":"2023-12-15T22:59:59.293952+0000","last_undegraded":"2023-12-15T22:59:59.293952+0000","last_fullsized":"2023-12-15T22:59:59.293952+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"209'186","reported_seq":545,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.295765+0000","last_change":"2023-12-15T22:59:57.298727+0000","last_active":"2023-12-15T22:59:59.295765+0000","last_peered":"2023-12-15T22:59:59.295765+0000","last_clean":"2023-12-15T22:59:59.295765+0000","last_became_active":"2023-12-15T22:56:39.557252+0000","last_became_peered":"2023-12-15T22:56:39.557252+0000","last_unstale":"2023-12-15T22:59:59.295765+0000","last_undegraded":"2023-12-15T22:59:59.295765+0000","last_fullsized":"2023-12-15T22:59:59.295765+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4453987,"num_objects":16,"num_object_clones":11,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":105,"num_read_kb":74518,"num_write":85,"num_write_kb":23945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":311,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.543326+0000","last_change":"2023-12-15T22:56:33.632348+0000","last_active":"2023-12-15T22:59:59.543326+0000","last_peered":"2023-12-15T22:59:59.543326+0000","last_clean":"2023-12-15T22:59:59.543326+0000","last_became_active":"2023-12-15T22:56:33.629022+0000","last_became_peered":"2023-12-15T22:56:33.629022+0000","last_unstale":"2023-12-15T22:59:59.543326+0000","last_undegraded":"2023-12-15T22:59:59.543326+0000","last_fullsized":"2023-12-15T22:59:59.543326+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"187'72","reported_seq":442,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.543210+0000","last_change":"2023-12-15T22:59:58.327213+0000","last_active":"2023-12-15T22:59:59.543210+0000","last_peered":"2023-12-15T22:59:59.543210+0000","last_clean":"2023-12-15T22:59:59.543210+0000","last_became_active":"2023-12-15T22:56:39.535145+0000","last_became_peered":"2023-12-15T22:56:39.535145+0000","last_unstale":"2023-12-15T22:59:59.543210+0000","last_undegraded":"2023-12-15T22:59:59.543210+0000","last_fullsized":"2023-12-15T22:59:59.543210+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3764801,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":31,"num_read_kb":29392,"num_write":27,"num_write_kb":8581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":309,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.302126+0000","last_change":"2023-12-15T22:56:33.627005+0000","last_active":"2023-12-15T22:59:59.302126+0000","last_peered":"2023-12-15T22:59:59.302126+0000","last_clean":"2023-12-15T22:59:59.302126+0000","last_became_active":"2023-12-15T22:56:33.622181+0000","last_became_peered":"2023-12-15T22:56:33.622181+0000","last_unstale":"2023-12-15T22:59:59.302126+0000","last_undegraded":"2023-12-15T22:59:59.302126+0000","last_fullsized":"2023-12-15T22:59:59.302126+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":91818324,"num_objects":279,"num_object_clones":200,"num_object_copies":837,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":46,"num_read":1590,"num_read_kb":1190640,"num_write":1273,"num_write_kb":370091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2885,"ondisk_log_size":2885,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":85792,"kb_used_data":50648,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94286048,"statfs":{"total":96636764160,"available":96548913152,"internally_reserved":0,"allocated":51863552,"data_stored":52450456,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":65604,"kb_used_data":34620,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94306236,"statfs":{"total":96636764160,"available":96569585664,"internally_reserved":0,"allocated":35450880,"data_stored":36124669,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":72784,"kb_used_data":41800,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94299056,"statfs":{"total":96636764160,"available":96562233344,"internally_reserved":0,"allocated":42803200,"data_stored":42181164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":82492,"kb_used_data":47348,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94289348,"statfs":{"total":96636764160,"available":96552292352,"internally_reserved":0,"allocated":48484352,"data_stored":49131488,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-15T23:00:04.608 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T23:00:04.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:04.747+0000 7f69a2d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:04.753 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:04.753+0000 7f69a2d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:04.926 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T23:00:04.927 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-12-15T23:00:04.939 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":317,"stamp":"2023-12-15T23:00:04.553068+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92408692,"num_objects":281,"num_object_clones":200,"num_object_copies":843,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":46,"num_read":1693,"num_read_kb":1190731,"num_write":1410,"num_write_kb":371985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2982,"ondisk_log_size":2982,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":306672,"kb_used_data":174416,"kb_used_omap":0,"kb_used_meta":132224,"kb_avail":377180688,"statfs":{"total":386547056640,"available":386233024512,"internally_reserved":0,"allocated":178601984,"data_stored":179887777,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":135397376},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3563117,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":25,"num_read_kb":25028,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000717"},"pg_stats":[{"pgid":"3.f","version":"185'90","reported_seq":495,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.348271+0000","last_change":"2023-12-15T22:59:57.312208+0000","last_active":"2023-12-15T22:59:59.348271+0000","last_peered":"2023-12-15T22:59:59.348271+0000","last_clean":"2023-12-15T22:59:59.348271+0000","last_became_active":"2023-12-15T22:56:39.503797+0000","last_became_peered":"2023-12-15T22:56:39.503797+0000","last_unstale":"2023-12-15T22:59:59.348271+0000","last_undegraded":"2023-12-15T22:59:59.348271+0000","last_fullsized":"2023-12-15T22:59:59.348271+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6991778,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":76,"num_read_kb":60668,"num_write":36,"num_write_kb":10667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":403,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.547183+0000","last_change":"2023-12-15T22:59:58.326705+0000","last_active":"2023-12-15T22:59:59.547183+0000","last_peered":"2023-12-15T22:59:59.547183+0000","last_clean":"2023-12-15T22:59:59.547183+0000","last_became_active":"2023-12-15T22:56:39.552856+0000","last_became_peered":"2023-12-15T22:56:39.552856+0000","last_unstale":"2023-12-15T22:59:59.547183+0000","last_undegraded":"2023-12-15T22:59:59.547183+0000","last_fullsized":"2023-12-15T22:59:59.547183+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"212'288","reported_seq":647,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.400744+0000","last_change":"2023-12-15T22:59:57.338044+0000","last_active":"2023-12-15T22:59:59.400744+0000","last_peered":"2023-12-15T22:59:59.400744+0000","last_clean":"2023-12-15T22:59:59.400744+0000","last_became_active":"2023-12-15T22:56:39.495490+0000","last_became_peered":"2023-12-15T22:56:39.495490+0000","last_unstale":"2023-12-15T22:59:59.400744+0000","last_undegraded":"2023-12-15T22:59:59.400744+0000","last_fullsized":"2023-12-15T22:59:59.400744+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":288,"log_dups_size":0,"ondisk_log_size":288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17227439,"num_objects":32,"num_object_clones":20,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":226,"num_read_kb":202764,"num_write":156,"num_write_kb":44891,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"208'291","reported_seq":625,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.325227+0000","last_change":"2023-12-15T22:59:57.330188+0000","last_active":"2023-12-15T22:59:59.325227+0000","last_peered":"2023-12-15T22:59:59.325227+0000","last_clean":"2023-12-15T22:59:59.325227+0000","last_became_active":"2023-12-15T22:56:39.579234+0000","last_became_peered":"2023-12-15T22:56:39.579234+0000","last_unstale":"2023-12-15T22:59:59.325227+0000","last_undegraded":"2023-12-15T22:59:59.325227+0000","last_fullsized":"2023-12-15T22:59:59.325227+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8891068,"num_objects":28,"num_object_clones":24,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":149,"num_read_kb":108446,"num_write":147,"num_write_kb":42462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"210'144","reported_seq":499,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.287440+0000","last_change":"2023-12-15T22:59:57.296538+0000","last_active":"2023-12-15T22:59:59.287440+0000","last_peered":"2023-12-15T22:59:59.287440+0000","last_clean":"2023-12-15T22:59:59.287440+0000","last_became_active":"2023-12-15T22:56:39.572176+0000","last_became_peered":"2023-12-15T22:56:39.572176+0000","last_unstale":"2023-12-15T22:59:59.287440+0000","last_undegraded":"2023-12-15T22:59:59.287440+0000","last_fullsized":"2023-12-15T22:59:59.287440+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3899257,"num_objects":14,"num_object_clones":10,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":58,"num_read_kb":29047,"num_write":60,"num_write_kb":18313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"209'141","reported_seq":496,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.322006+0000","last_change":"2023-12-15T22:59:57.320770+0000","last_active":"2023-12-15T22:59:59.322006+0000","last_peered":"2023-12-15T22:59:59.322006+0000","last_clean":"2023-12-15T22:59:59.322006+0000","last_became_active":"2023-12-15T22:56:39.516361+0000","last_became_peered":"2023-12-15T22:56:39.516361+0000","last_unstale":"2023-12-15T22:59:59.322006+0000","last_undegraded":"2023-12-15T22:59:59.322006+0000","last_fullsized":"2023-12-15T22:59:59.322006+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":51,"num_read_kb":38717,"num_write":51,"num_write_kb":14458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"204'552","reported_seq":735,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.322641+0000","last_change":"2023-12-15T22:59:57.320651+0000","last_active":"2023-12-15T22:59:59.322641+0000","last_peered":"2023-12-15T22:59:59.322641+0000","last_clean":"2023-12-15T22:59:59.322641+0000","last_became_active":"2023-12-15T22:56:39.519748+0000","last_became_peered":"2023-12-15T22:56:39.519748+0000","last_unstale":"2023-12-15T22:59:59.322641+0000","last_undegraded":"2023-12-15T22:59:59.322641+0000","last_fullsized":"2023-12-15T22:59:59.322641+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":552,"log_dups_size":0,"ondisk_log_size":552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13933267,"num_objects":50,"num_object_clones":39,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":235,"num_read_kb":149617,"num_write":217,"num_write_kb":62654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"200'133","reported_seq":515,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.305402+0000","last_change":"2023-12-15T22:59:57.321958+0000","last_active":"2023-12-15T22:59:59.305402+0000","last_peered":"2023-12-15T22:59:59.305402+0000","last_clean":"2023-12-15T22:59:59.305402+0000","last_became_active":"2023-12-15T22:56:39.543179+0000","last_became_peered":"2023-12-15T22:56:39.543179+0000","last_unstale":"2023-12-15T22:59:59.305402+0000","last_undegraded":"2023-12-15T22:59:59.305402+0000","last_fullsized":"2023-12-15T22:59:59.305402+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7361565,"num_objects":13,"num_object_clones":9,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":116,"num_read_kb":103738,"num_write":70,"num_write_kb":18597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":452,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.294380+0000","last_change":"2023-12-15T22:56:32.607017+0000","last_active":"2023-12-15T22:59:59.294380+0000","last_peered":"2023-12-15T22:59:59.294380+0000","last_clean":"2023-12-15T22:59:59.294380+0000","last_became_active":"2023-12-15T22:56:32.602384+0000","last_became_peered":"2023-12-15T22:56:32.602384+0000","last_unstale":"2023-12-15T22:59:59.294380+0000","last_undegraded":"2023-12-15T22:59:59.294380+0000","last_fullsized":"2023-12-15T22:59:59.294380+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_clean_scrub_stamp":"2023-12-15T22:56:31.525572+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":309,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.303877+0000","last_change":"2023-12-15T22:56:33.627481+0000","last_active":"2023-12-15T22:59:59.303877+0000","last_peered":"2023-12-15T22:59:59.303877+0000","last_clean":"2023-12-15T22:59:59.303877+0000","last_became_active":"2023-12-15T22:56:33.624689+0000","last_became_peered":"2023-12-15T22:56:33.624689+0000","last_unstale":"2023-12-15T22:59:59.303877+0000","last_undegraded":"2023-12-15T22:59:59.303877+0000","last_fullsized":"2023-12-15T22:59:59.303877+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"208'178","reported_seq":557,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.309217+0000","last_change":"2023-12-15T22:59:57.304470+0000","last_active":"2023-12-15T22:59:59.309217+0000","last_peered":"2023-12-15T22:59:59.309217+0000","last_clean":"2023-12-15T22:59:59.309217+0000","last_became_active":"2023-12-15T22:56:39.529359+0000","last_became_peered":"2023-12-15T22:56:39.529359+0000","last_unstale":"2023-12-15T22:59:59.309217+0000","last_undegraded":"2023-12-15T22:59:59.309217+0000","last_fullsized":"2023-12-15T22:59:59.309217+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6941489,"num_objects":19,"num_object_clones":11,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":137,"num_read_kb":97172,"num_write":100,"num_write_kb":28930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":337,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.320394+0000","last_change":"2023-12-15T22:56:33.655797+0000","last_active":"2023-12-15T22:59:59.320394+0000","last_peered":"2023-12-15T22:59:59.320394+0000","last_clean":"2023-12-15T22:59:59.320394+0000","last_became_active":"2023-12-15T22:56:33.652885+0000","last_became_peered":"2023-12-15T22:56:33.652885+0000","last_unstale":"2023-12-15T22:59:59.320394+0000","last_undegraded":"2023-12-15T22:59:59.320394+0000","last_fullsized":"2023-12-15T22:59:59.320394+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"208'254","reported_seq":576,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.321298+0000","last_change":"2023-12-15T22:59:57.298807+0000","last_active":"2023-12-15T22:59:59.321298+0000","last_peered":"2023-12-15T22:59:59.321298+0000","last_clean":"2023-12-15T22:59:59.321298+0000","last_became_active":"2023-12-15T22:56:39.543509+0000","last_became_peered":"2023-12-15T22:56:39.543509+0000","last_unstale":"2023-12-15T22:59:59.321298+0000","last_undegraded":"2023-12-15T22:59:59.321298+0000","last_fullsized":"2023-12-15T22:59:59.321298+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":254,"log_dups_size":0,"ondisk_log_size":254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4386759,"num_objects":25,"num_object_clones":17,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":145,"num_read_kb":111254,"num_write":98,"num_write_kb":29596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":309,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.309106+0000","last_change":"2023-12-15T22:56:33.600485+0000","last_active":"2023-12-15T22:59:59.309106+0000","last_peered":"2023-12-15T22:59:59.309106+0000","last_clean":"2023-12-15T22:59:59.309106+0000","last_became_active":"2023-12-15T22:56:33.597320+0000","last_became_peered":"2023-12-15T22:56:33.597320+0000","last_unstale":"2023-12-15T22:59:59.309106+0000","last_undegraded":"2023-12-15T22:59:59.309106+0000","last_fullsized":"2023-12-15T22:59:59.309106+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"200'120","reported_seq":488,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.309031+0000","last_change":"2023-12-15T22:59:57.297889+0000","last_active":"2023-12-15T22:59:59.309031+0000","last_peered":"2023-12-15T22:59:59.309031+0000","last_clean":"2023-12-15T22:59:59.309031+0000","last_became_active":"2023-12-15T22:56:39.560980+0000","last_became_peered":"2023-12-15T22:56:39.560980+0000","last_unstale":"2023-12-15T22:59:59.309031+0000","last_undegraded":"2023-12-15T22:59:59.309031+0000","last_fullsized":"2023-12-15T22:59:59.309031+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6134654,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":62,"num_read_kb":52203,"num_write":49,"num_write_kb":14551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":337,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.323730+0000","last_change":"2023-12-15T22:56:33.648541+0000","last_active":"2023-12-15T22:59:59.323730+0000","last_peered":"2023-12-15T22:59:59.323730+0000","last_clean":"2023-12-15T22:59:59.323730+0000","last_became_active":"2023-12-15T22:56:33.643681+0000","last_became_peered":"2023-12-15T22:56:33.643681+0000","last_unstale":"2023-12-15T22:59:59.323730+0000","last_undegraded":"2023-12-15T22:59:59.323730+0000","last_fullsized":"2023-12-15T22:59:59.323730+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"211'205","reported_seq":532,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.541673+0000","last_change":"2023-12-15T22:59:58.387917+0000","last_active":"2023-12-15T22:59:59.541673+0000","last_peered":"2023-12-15T22:59:59.541673+0000","last_clean":"2023-12-15T22:59:59.541673+0000","last_became_active":"2023-12-15T22:56:39.506510+0000","last_became_peered":"2023-12-15T22:56:39.506510+0000","last_unstale":"2023-12-15T22:59:59.541673+0000","last_undegraded":"2023-12-15T22:59:59.541673+0000","last_fullsized":"2023-12-15T22:59:59.541673+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5361565,"num_objects":22,"num_object_clones":15,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":111,"num_read_kb":75801,"num_write":83,"num_write_kb":24115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"185'98","reported_seq":441,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.302204+0000","last_change":"2023-12-15T22:59:57.336227+0000","last_active":"2023-12-15T22:59:59.302204+0000","last_peered":"2023-12-15T22:59:59.302204+0000","last_clean":"2023-12-15T22:59:59.302204+0000","last_became_active":"2023-12-15T22:56:39.511159+0000","last_became_peered":"2023-12-15T22:56:39.511159+0000","last_unstale":"2023-12-15T22:59:59.302204+0000","last_undegraded":"2023-12-15T22:59:59.302204+0000","last_fullsized":"2023-12-15T22:59:59.302204+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":21,"num_read_kb":7340,"num_write":37,"num_write_kb":11102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":311,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.545607+0000","last_change":"2023-12-15T22:56:33.633890+0000","last_active":"2023-12-15T22:59:59.545607+0000","last_peered":"2023-12-15T22:59:59.545607+0000","last_clean":"2023-12-15T22:59:59.545607+0000","last_became_active":"2023-12-15T22:56:33.631134+0000","last_became_peered":"2023-12-15T22:56:33.631134+0000","last_unstale":"2023-12-15T22:59:59.545607+0000","last_undegraded":"2023-12-15T22:59:59.545607+0000","last_fullsized":"2023-12-15T22:59:59.545607+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"204'133","reported_seq":484,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.543977+0000","last_change":"2023-12-15T22:59:58.326883+0000","last_active":"2023-12-15T22:59:59.543977+0000","last_peered":"2023-12-15T22:59:59.543977+0000","last_clean":"2023-12-15T22:59:59.543977+0000","last_became_active":"2023-12-15T22:56:39.532730+0000","last_became_peered":"2023-12-15T22:56:39.532730+0000","last_unstale":"2023-12-15T22:59:59.543977+0000","last_undegraded":"2023-12-15T22:59:59.543977+0000","last_fullsized":"2023-12-15T22:59:59.543977+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2470695,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":67,"num_read_kb":49963,"num_write":57,"num_write_kb":17229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":323,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.293952+0000","last_change":"2023-12-15T22:56:33.655736+0000","last_active":"2023-12-15T22:59:59.293952+0000","last_peered":"2023-12-15T22:59:59.293952+0000","last_clean":"2023-12-15T22:59:59.293952+0000","last_became_active":"2023-12-15T22:56:33.652917+0000","last_became_peered":"2023-12-15T22:56:33.652917+0000","last_unstale":"2023-12-15T22:59:59.293952+0000","last_undegraded":"2023-12-15T22:59:59.293952+0000","last_fullsized":"2023-12-15T22:59:59.293952+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"209'186","reported_seq":545,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.295765+0000","last_change":"2023-12-15T22:59:57.298727+0000","last_active":"2023-12-15T22:59:59.295765+0000","last_peered":"2023-12-15T22:59:59.295765+0000","last_clean":"2023-12-15T22:59:59.295765+0000","last_became_active":"2023-12-15T22:56:39.557252+0000","last_became_peered":"2023-12-15T22:56:39.557252+0000","last_unstale":"2023-12-15T22:59:59.295765+0000","last_undegraded":"2023-12-15T22:59:59.295765+0000","last_fullsized":"2023-12-15T22:59:59.295765+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4453987,"num_objects":16,"num_object_clones":11,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":105,"num_read_kb":74518,"num_write":85,"num_write_kb":23945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":311,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.543326+0000","last_change":"2023-12-15T22:56:33.632348+0000","last_active":"2023-12-15T22:59:59.543326+0000","last_peered":"2023-12-15T22:59:59.543326+0000","last_clean":"2023-12-15T22:59:59.543326+0000","last_became_active":"2023-12-15T22:56:33.629022+0000","last_became_peered":"2023-12-15T22:56:33.629022+0000","last_unstale":"2023-12-15T22:59:59.543326+0000","last_undegraded":"2023-12-15T22:59:59.543326+0000","last_fullsized":"2023-12-15T22:59:59.543326+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"187'72","reported_seq":442,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.543210+0000","last_change":"2023-12-15T22:59:58.327213+0000","last_active":"2023-12-15T22:59:59.543210+0000","last_peered":"2023-12-15T22:59:59.543210+0000","last_clean":"2023-12-15T22:59:59.543210+0000","last_became_active":"2023-12-15T22:56:39.535145+0000","last_became_peered":"2023-12-15T22:56:39.535145+0000","last_unstale":"2023-12-15T22:59:59.543210+0000","last_undegraded":"2023-12-15T22:59:59.543210+0000","last_fullsized":"2023-12-15T22:59:59.543210+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3764801,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":31,"num_read_kb":29392,"num_write":27,"num_write_kb":8581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":309,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-15T22:59:59.302126+0000","last_change":"2023-12-15T22:56:33.627005+0000","last_active":"2023-12-15T22:59:59.302126+0000","last_peered":"2023-12-15T22:59:59.302126+0000","last_clean":"2023-12-15T22:59:59.302126+0000","last_became_active":"2023-12-15T22:56:33.622181+0000","last_became_peered":"2023-12-15T22:56:33.622181+0000","last_unstale":"2023-12-15T22:59:59.302126+0000","last_undegraded":"2023-12-15T22:59:59.302126+0000","last_fullsized":"2023-12-15T22:59:59.302126+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":91818324,"num_objects":279,"num_object_clones":200,"num_object_copies":837,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":46,"num_read":1590,"num_read_kb":1190640,"num_write":1273,"num_write_kb":370091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2885,"ondisk_log_size":2885,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":85792,"kb_used_data":50648,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94286048,"statfs":{"total":96636764160,"available":96548913152,"internally_reserved":0,"allocated":51863552,"data_stored":52450456,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":65604,"kb_used_data":34620,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94306236,"statfs":{"total":96636764160,"available":96569585664,"internally_reserved":0,"allocated":35450880,"data_stored":36124669,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":72784,"kb_used_data":41800,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94299056,"statfs":{"total":96636764160,"available":96562233344,"internally_reserved":0,"allocated":42803200,"data_stored":42181164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":82492,"kb_used_data":47348,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94289348,"statfs":{"total":96636764160,"available":96552292352,"internally_reserved":0,"allocated":48484352,"data_stored":49131488,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-15T23:00:04.940 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2023-12-15T23:00:04.940 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T23:00:04.940 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-12-15T23:00:04.940 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-12-15T23:00:05.045 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:05.041+0000 7f2e1bc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:05.049 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:05.048+0000 7f2e1bc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:05.060 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T23:00:05.060 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T23:00:05.060 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T23:00:05.060 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T23:00:05.060 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T23:00:05.061 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T23:00:05.061 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T23:00:05.061 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T23:00:05.061 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T23:00:05.061 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T23:00:05.061 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T23:00:05.061 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T23:00:05.063 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T23:00:05.064 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:00:05.064 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-12-15T23:00:05.169 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:05.165+0000 7f4ac562d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:05.172 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:05.172+0000 7f4ac562d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:05.184 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T23:00:05.184 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T23:00:05.184 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T23:00:05.184 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T23:00:05.184 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T23:00:05.185 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T23:00:05.185 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T23:00:05.185 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T23:00:05.185 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T23:00:05.185 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T23:00:05.185 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T23:00:05.185 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T23:00:05.187 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T23:00:05.187 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:00:05.187 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-12-15T23:00:05.293 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:05.289+0000 7fb23fe6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:05.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:05.297+0000 7fb23fe6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:05.310 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T23:00:05.310 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T23:00:05.310 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T23:00:05.310 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T23:00:05.310 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T23:00:05.310 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T23:00:05.311 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T23:00:05.311 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T23:00:05.311 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T23:00:05.311 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T23:00:05.311 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T23:00:05.311 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T23:00:05.313 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T23:00:05.313 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:00:05.313 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-12-15T23:00:05.316 INFO:tasks.rados.rados.0.smithi071.stdout:881: done (1 left) 2023-12-15T23:00:05.316 INFO:tasks.rados.rados.0.smithi071.stdout:882: write (excl) oid 42 current snap is 100 2023-12-15T23:00:05.316 INFO:tasks.rados.rados.0.smithi071.stdout:879: finishing write tid 1 to smithi07183189-48 2023-12-15T23:00:05.316 INFO:tasks.rados.rados.0.smithi071.stdout:882: seq_num 428 ranges {793396=722340,2128999=690812} 2023-12-15T23:00:05.326 INFO:tasks.rados.rados.0.smithi071.stdout:882: writing smithi07183189-42 from 793396 to 1515736 tid 1 2023-12-15T23:00:05.335 INFO:tasks.rados.rados.0.smithi071.stdout:882: writing smithi07183189-42 from 2128999 to 2819811 tid 2 2023-12-15T23:00:05.335 INFO:tasks.rados.rados.0.smithi071.stdout:883: snap_remove snap 98 2023-12-15T23:00:05.429 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:05.425+0000 7fbbcff5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:05.433 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:05.433+0000 7fbbcff5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:05.446 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T23:00:05.446 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T23:00:05.446 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T23:00:05.446 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T23:00:05.446 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T23:00:05.446 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T23:00:05.447 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T23:00:05.447 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T23:00:05.447 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T23:00:05.447 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T23:00:05.447 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T23:00:05.447 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T23:00:05.449 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T23:00:05.449 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:00:06.060 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T23:00:06.171 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:06.170+0000 7f931f8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:06.174 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:06.174+0000 7f931f8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:06.321 INFO:tasks.rados.rados.0.smithi071.stdout:883: done (2 left) 2023-12-15T23:00:06.321 INFO:tasks.rados.rados.0.smithi071.stdout:884: write (excl) oid 22 current snap is 100 2023-12-15T23:00:06.321 INFO:tasks.rados.rados.0.smithi071.stdout:879: finishing write tid 2 to smithi07183189-48 2023-12-15T23:00:06.321 INFO:tasks.rados.rados.0.smithi071.stdout:884: seq_num 429 ranges {410203=797589,1787481=434470,2681405=528798} 2023-12-15T23:00:06.327 INFO:tasks.rados.rados.0.smithi071.stdout:884: writing smithi07183189-22 from 410203 to 1207792 tid 1 2023-12-15T23:00:06.332 INFO:tasks.rados.rados.0.smithi071.stdout:884: writing smithi07183189-22 from 1787481 to 2221951 tid 2 2023-12-15T23:00:06.338 INFO:tasks.rados.rados.0.smithi071.stdout:884: writing smithi07183189-22 from 2681405 to 3210203 tid 3 2023-12-15T23:00:06.338 INFO:tasks.rados.rados.0.smithi071.stdout:885: delete oid 16 current snap is 100 2023-12-15T23:00:06.338 INFO:tasks.rados.rados.0.smithi071.stdout:879: finishing write tid 3 to smithi07183189-48 2023-12-15T23:00:06.338 INFO:tasks.rados.rados.0.smithi071.stdout:879: finishing write tid 4 to smithi07183189-48 2023-12-15T23:00:06.339 INFO:tasks.rados.rados.0.smithi071.stdout:879: oid 48 updating version 0 to 192 2023-12-15T23:00:06.339 INFO:tasks.rados.rados.0.smithi071.stdout:879: oid 48 version 192 is already newer than 191 2023-12-15T23:00:06.339 INFO:tasks.rados.rados.0.smithi071.stdout:879: oid 48 updating version 192 to 193 2023-12-15T23:00:06.339 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 193 (ObjNum 426 snap 100 seq_num 426) dirty exists 2023-12-15T23:00:06.339 INFO:tasks.rados.rados.0.smithi071.stdout:879: left oid 48 (ObjNum 426 snap 100 seq_num 426) 2023-12-15T23:00:06.339 INFO:tasks.rados.rados.0.smithi071.stdout:882: finishing write tid 1 to smithi07183189-42 2023-12-15T23:00:06.339 INFO:tasks.rados.rados.0.smithi071.stdout:882: finishing write tid 2 to smithi07183189-42 2023-12-15T23:00:06.339 INFO:tasks.rados.rados.0.smithi071.stdout:882: finishing write tid 3 to smithi07183189-42 2023-12-15T23:00:06.339 INFO:tasks.rados.rados.0.smithi071.stdout:882: finishing write tid 4 to smithi07183189-42 2023-12-15T23:00:06.339 INFO:tasks.rados.rados.0.smithi071.stdout:882: oid 42 updating version 0 to 198 2023-12-15T23:00:06.339 INFO:tasks.rados.rados.0.smithi071.stdout:882: oid 42 version 198 is already newer than 197 2023-12-15T23:00:06.339 INFO:tasks.rados.rados.0.smithi071.stdout:882: oid 42 updating version 198 to 199 2023-12-15T23:00:06.339 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 42 v 199 (ObjNum 428 snap 100 seq_num 428) dirty exists 2023-12-15T23:00:06.339 INFO:tasks.rados.rados.0.smithi071.stdout:882: left oid 42 (ObjNum 428 snap 100 seq_num 428) 2023-12-15T23:00:06.386 INFO:tasks.rados.rados.0.smithi071.stdout:879: done (3 left) 2023-12-15T23:00:06.386 INFO:tasks.rados.rados.0.smithi071.stdout:882: done (2 left) 2023-12-15T23:00:06.386 INFO:tasks.rados.rados.0.smithi071.stdout:885: done (1 left) 2023-12-15T23:00:06.386 INFO:tasks.rados.rados.0.smithi071.stdout:886: write (excl) oid 15 current snap is 100 2023-12-15T23:00:06.386 INFO:tasks.rados.rados.0.smithi071.stdout:886: seq_num 431 ranges {443817=548087,1788092=438139,3019898=223919} 2023-12-15T23:00:06.393 INFO:tasks.rados.rados.0.smithi071.stdout:886: writing smithi07183189-15 from 443817 to 991904 tid 1 2023-12-15T23:00:06.402 INFO:tasks.rados.rados.0.smithi071.stdout:886: writing smithi07183189-15 from 1788092 to 2226231 tid 2 2023-12-15T23:00:06.409 INFO:tasks.rados.rados.0.smithi071.stdout:886: writing smithi07183189-15 from 3019898 to 3243817 tid 3 2023-12-15T23:00:06.409 INFO:tasks.rados.rados.0.smithi071.stdout:887: write oid 33 current snap is 100 2023-12-15T23:00:06.409 INFO:tasks.rados.rados.0.smithi071.stdout:884: finishing write tid 1 to smithi07183189-22 2023-12-15T23:00:06.409 INFO:tasks.rados.rados.0.smithi071.stdout:887: seq_num 432 ranges {460624=623336,1846574=698150,3163674=96950} 2023-12-15T23:00:06.417 INFO:tasks.rados.rados.0.smithi071.stdout:887: writing smithi07183189-33 from 460624 to 1083960 tid 1 2023-12-15T23:00:06.427 INFO:tasks.rados.rados.0.smithi071.stdout:887: writing smithi07183189-33 from 1846574 to 2544724 tid 2 2023-12-15T23:00:06.432 INFO:tasks.rados.rados.0.smithi071.stdout:887: writing smithi07183189-33 from 3163674 to 3260624 tid 3 2023-12-15T23:00:06.432 INFO:tasks.rados.rados.0.smithi071.stdout:888: delete oid 46 current snap is 100 2023-12-15T23:00:06.432 INFO:tasks.rados.rados.0.smithi071.stdout:884: finishing write tid 2 to smithi07183189-22 2023-12-15T23:00:06.476 INFO:tasks.rados.rados.0.smithi071.stdout:884: finishing write tid 3 to smithi07183189-22 2023-12-15T23:00:06.493 INFO:tasks.rados.rados.0.smithi071.stdout:888: done (3 left) 2023-12-15T23:00:06.493 INFO:tasks.rados.rados.0.smithi071.stdout:889: write oid 48 current snap is 100 2023-12-15T23:00:06.493 INFO:tasks.rados.rados.0.smithi071.stdout:889: seq_num 434 ranges {494238=490187,1563538=701819,2818520=475718} 2023-12-15T23:00:06.500 INFO:tasks.rados.rados.0.smithi071.stdout:889: writing smithi07183189-48 from 494238 to 984425 tid 1 2023-12-15T23:00:06.509 INFO:tasks.rados.rados.0.smithi071.stdout:889: writing smithi07183189-48 from 1563538 to 2265357 tid 2 2023-12-15T23:00:06.516 INFO:tasks.rados.rados.0.smithi071.stdout:889: writing smithi07183189-48 from 2818520 to 3294238 tid 3 2023-12-15T23:00:06.516 INFO:tasks.rados.rados.0.smithi071.stdout:890: write (excl) oid 21 current snap is 100 2023-12-15T23:00:06.517 INFO:tasks.rados.rados.0.smithi071.stdout:890: seq_num 435 ranges {511045=565436,1622020=445477,2729590=556855} 2023-12-15T23:00:06.524 INFO:tasks.rados.rados.0.smithi071.stdout:890: writing smithi07183189-21 from 511045 to 1076481 tid 1 2023-12-15T23:00:06.531 INFO:tasks.rados.rados.0.smithi071.stdout:890: writing smithi07183189-21 from 1622020 to 2067497 tid 2 2023-12-15T23:00:06.540 INFO:tasks.rados.rados.0.smithi071.stdout:890: writing smithi07183189-21 from 2729590 to 3286445 tid 3 2023-12-15T23:00:06.540 INFO:tasks.rados.rados.0.smithi071.stdout:891: read oid 50 snap -1 2023-12-15T23:00:06.540 INFO:tasks.rados.rados.0.smithi071.stdout:891: expect (ObjNum 353 snap 88 seq_num 353) 2023-12-15T23:00:06.540 INFO:tasks.rados.rados.0.smithi071.stdout:892: read oid 18 snap -1 2023-12-15T23:00:06.540 INFO:tasks.rados.rados.0.smithi071.stdout:892: expect deleted 2023-12-15T23:00:06.540 INFO:tasks.rados.rados.0.smithi071.stdout:893: write oid 2 current snap is 100 2023-12-15T23:00:06.541 INFO:tasks.rados.rados.0.smithi071.stdout:886: finishing write tid 1 to smithi07183189-15 2023-12-15T23:00:06.541 INFO:tasks.rados.rados.0.smithi071.stdout:893: seq_num 436 ranges {527852=640685,1680502=589135,2757013=570839} 2023-12-15T23:00:06.548 INFO:tasks.rados.rados.0.smithi071.stdout:893: writing smithi07183189-2 from 527852 to 1168537 tid 1 2023-12-15T23:00:06.557 INFO:tasks.rados.rados.0.smithi071.stdout:893: writing smithi07183189-2 from 1680502 to 2269637 tid 2 2023-12-15T23:00:06.565 INFO:tasks.rados.rados.0.smithi071.stdout:893: writing smithi07183189-2 from 2757013 to 3327852 tid 3 2023-12-15T23:00:06.565 INFO:tasks.rados.rados.0.smithi071.stdout:894: read oid 47 snap -1 2023-12-15T23:00:06.565 INFO:tasks.rados.rados.0.smithi071.stdout:894: expect (ObjNum 423 snap 99 seq_num 423) 2023-12-15T23:00:06.565 INFO:tasks.rados.rados.0.smithi071.stdout:895: snap_create 2023-12-15T23:00:06.566 INFO:tasks.rados.rados.0.smithi071.stdout:884: finishing write tid 4 to smithi07183189-22 2023-12-15T23:00:06.566 INFO:tasks.rados.rados.0.smithi071.stdout:887: finishing write tid 1 to smithi07183189-33 2023-12-15T23:00:06.566 INFO:tasks.rados.rados.0.smithi071.stdout:884: finishing write tid 5 to smithi07183189-22 2023-12-15T23:00:06.566 INFO:tasks.rados.rados.0.smithi071.stdout:884: oid 22 updating version 0 to 214 2023-12-15T23:00:06.566 INFO:tasks.rados.rados.0.smithi071.stdout:884: oid 22 version 214 is already newer than 213 2023-12-15T23:00:06.566 INFO:tasks.rados.rados.0.smithi071.stdout:884: oid 22 version 214 is already newer than 212 2023-12-15T23:00:06.566 INFO:tasks.rados.rados.0.smithi071.stdout:884: oid 22 updating version 214 to 215 2023-12-15T23:00:06.566 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 22 v 215 (ObjNum 429 snap 100 seq_num 429) dirty exists 2023-12-15T23:00:06.566 INFO:tasks.rados.rados.0.smithi071.stdout:884: left oid 22 (ObjNum 429 snap 100 seq_num 429) 2023-12-15T23:00:06.576 INFO:tasks.rados.rados.0.smithi071.stdout:887: finishing write tid 2 to smithi07183189-33 2023-12-15T23:00:06.592 INFO:tasks.rados.rados.0.smithi071.stdout:886: finishing write tid 2 to smithi07183189-15 2023-12-15T23:00:06.616 INFO:tasks.rados.rados.0.smithi071.stdout:894: expect (ObjNum 423 snap 99 seq_num 423) 2023-12-15T23:00:06.755 INFO:tasks.rados.rados.0.smithi071.stdout:890: finishing write tid 1 to smithi07183189-21 2023-12-15T23:00:06.755 INFO:tasks.rados.rados.0.smithi071.stdout:887: finishing write tid 3 to smithi07183189-33 2023-12-15T23:00:06.755 INFO:tasks.rados.rados.0.smithi071.stdout:893: finishing write tid 1 to smithi07183189-2 2023-12-15T23:00:06.755 INFO:tasks.rados.rados.0.smithi071.stdout:886: finishing write tid 3 to smithi07183189-15 2023-12-15T23:00:06.755 INFO:tasks.rados.rados.0.smithi071.stdout:890: finishing write tid 2 to smithi07183189-21 2023-12-15T23:00:06.756 INFO:tasks.rados.rados.0.smithi071.stdout:886: finishing write tid 4 to smithi07183189-15 2023-12-15T23:00:06.756 INFO:tasks.rados.rados.0.smithi071.stdout:886: finishing write tid 5 to smithi07183189-15 2023-12-15T23:00:06.756 INFO:tasks.rados.rados.0.smithi071.stdout:886: oid 15 updating version 0 to 205 2023-12-15T23:00:06.756 INFO:tasks.rados.rados.0.smithi071.stdout:886: oid 15 version 205 is already newer than 204 2023-12-15T23:00:06.756 INFO:tasks.rados.rados.0.smithi071.stdout:886: oid 15 version 205 is already newer than 203 2023-12-15T23:00:06.756 INFO:tasks.rados.rados.0.smithi071.stdout:886: oid 15 version 205 is already newer than 202 2023-12-15T23:00:06.756 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 205 (ObjNum 431 snap 100 seq_num 431) dirty exists 2023-12-15T23:00:06.756 INFO:tasks.rados.rados.0.smithi071.stdout:886: left oid 15 (ObjNum 431 snap 100 seq_num 431) 2023-12-15T23:00:06.756 INFO:tasks.rados.rados.0.smithi071.stdout:887: finishing write tid 4 to smithi07183189-33 2023-12-15T23:00:06.756 INFO:tasks.rados.rados.0.smithi071.stdout:887: finishing write tid 5 to smithi07183189-33 2023-12-15T23:00:06.756 INFO:tasks.rados.rados.0.smithi071.stdout:887: oid 33 updating version 0 to 572 2023-12-15T23:00:06.756 INFO:tasks.rados.rados.0.smithi071.stdout:887: oid 33 updating version 572 to 574 2023-12-15T23:00:06.756 INFO:tasks.rados.rados.0.smithi071.stdout:887: oid 33 updating version 574 to 575 2023-12-15T23:00:06.756 INFO:tasks.rados.rados.0.smithi071.stdout:887: oid 33 version 575 is already newer than 573 2023-12-15T23:00:06.756 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 33 v 575 (ObjNum 432 snap 100 seq_num 432) dirty exists 2023-12-15T23:00:06.757 INFO:tasks.rados.rados.0.smithi071.stdout:887: left oid 33 (ObjNum 432 snap 100 seq_num 432) 2023-12-15T23:00:06.757 INFO:tasks.rados.rados.0.smithi071.stdout:891: expect (ObjNum 353 snap 88 seq_num 353) 2023-12-15T23:00:07.106 INFO:tasks.rados.rados.0.smithi071.stdout:893: finishing write tid 2 to smithi07183189-2 2023-12-15T23:00:07.106 INFO:tasks.rados.rados.0.smithi071.stdout:890: finishing write tid 3 to smithi07183189-21 2023-12-15T23:00:07.106 INFO:tasks.rados.rados.0.smithi071.stdout:889: finishing write tid 1 to smithi07183189-48 2023-12-15T23:00:07.107 INFO:tasks.rados.rados.0.smithi071.stdout:889: finishing write tid 2 to smithi07183189-48 2023-12-15T23:00:07.107 INFO:tasks.rados.rados.0.smithi071.stdout:889: finishing write tid 3 to smithi07183189-48 2023-12-15T23:00:07.107 INFO:tasks.rados.rados.0.smithi071.stdout:890: finishing write tid 4 to smithi07183189-21 2023-12-15T23:00:07.107 INFO:tasks.rados.rados.0.smithi071.stdout:889: finishing write tid 4 to smithi07183189-48 2023-12-15T23:00:07.107 INFO:tasks.rados.rados.0.smithi071.stdout:890: finishing write tid 5 to smithi07183189-21 2023-12-15T23:00:07.107 INFO:tasks.rados.rados.0.smithi071.stdout:890: oid 21 updating version 0 to 259 2023-12-15T23:00:07.107 INFO:tasks.rados.rados.0.smithi071.stdout:890: oid 21 updating version 259 to 262 2023-12-15T23:00:07.107 INFO:tasks.rados.rados.0.smithi071.stdout:890: oid 21 version 262 is already newer than 260 2023-12-15T23:00:07.107 INFO:tasks.rados.rados.0.smithi071.stdout:890: oid 21 version 262 is already newer than 261 2023-12-15T23:00:07.107 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 21 v 262 (ObjNum 435 snap 100 seq_num 435) dirty exists 2023-12-15T23:00:07.107 INFO:tasks.rados.rados.0.smithi071.stdout:890: left oid 21 (ObjNum 435 snap 100 seq_num 435) 2023-12-15T23:00:07.107 INFO:tasks.rados.rados.0.smithi071.stdout:889: finishing write tid 5 to smithi07183189-48 2023-12-15T23:00:07.107 INFO:tasks.rados.rados.0.smithi071.stdout:889: oid 48 updating version 0 to 209 2023-12-15T23:00:07.107 INFO:tasks.rados.rados.0.smithi071.stdout:889: oid 48 version 209 is already newer than 209 2023-12-15T23:00:07.107 INFO:tasks.rados.rados.0.smithi071.stdout:889: oid 48 version 209 is already newer than 209 2023-12-15T23:00:07.108 INFO:tasks.rados.rados.0.smithi071.stdout:889: oid 48 version 209 is already newer than 209 2023-12-15T23:00:07.108 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 209 (ObjNum 434 snap 100 seq_num 434) dirty exists 2023-12-15T23:00:07.108 INFO:tasks.rados.rados.0.smithi071.stdout:889: left oid 48 (ObjNum 434 snap 100 seq_num 434) 2023-12-15T23:00:07.108 INFO:tasks.rados.rados.0.smithi071.stdout:893: finishing write tid 3 to smithi07183189-2 2023-12-15T23:00:07.108 INFO:tasks.rados.rados.0.smithi071.stdout:893: finishing write tid 4 to smithi07183189-2 2023-12-15T23:00:07.108 INFO:tasks.rados.rados.0.smithi071.stdout:893: finishing write tid 5 to smithi07183189-2 2023-12-15T23:00:07.108 INFO:tasks.rados.rados.0.smithi071.stdout:893: oid 2 updating version 0 to 142 2023-12-15T23:00:07.108 INFO:tasks.rados.rados.0.smithi071.stdout:893: oid 2 updating version 142 to 145 2023-12-15T23:00:07.108 INFO:tasks.rados.rados.0.smithi071.stdout:893: oid 2 version 145 is already newer than 143 2023-12-15T23:00:07.108 INFO:tasks.rados.rados.0.smithi071.stdout:893: oid 2 version 145 is already newer than 145 2023-12-15T23:00:07.108 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 145 (ObjNum 436 snap 100 seq_num 436) dirty exists 2023-12-15T23:00:07.108 INFO:tasks.rados.rados.0.smithi071.stdout:893: left oid 2 (ObjNum 436 snap 100 seq_num 436) 2023-12-15T23:00:07.332 INFO:tasks.rados.rados.0.smithi071.stdout:884: done (9 left) 2023-12-15T23:00:07.332 INFO:tasks.rados.rados.0.smithi071.stdout:886: done (8 left) 2023-12-15T23:00:07.332 INFO:tasks.rados.rados.0.smithi071.stdout:887: done (7 left) 2023-12-15T23:00:07.332 INFO:tasks.rados.rados.0.smithi071.stdout:889: done (6 left) 2023-12-15T23:00:07.333 INFO:tasks.rados.rados.0.smithi071.stdout:890: done (5 left) 2023-12-15T23:00:07.333 INFO:tasks.rados.rados.0.smithi071.stdout:891: done (4 left) 2023-12-15T23:00:07.333 INFO:tasks.rados.rados.0.smithi071.stdout:892: done (3 left) 2023-12-15T23:00:07.333 INFO:tasks.rados.rados.0.smithi071.stdout:893: done (2 left) 2023-12-15T23:00:07.333 INFO:tasks.rados.rados.0.smithi071.stdout:894: done (1 left) 2023-12-15T23:00:07.333 INFO:tasks.rados.rados.0.smithi071.stdout:895: done (0 left) 2023-12-15T23:00:07.333 INFO:tasks.rados.rados.0.smithi071.stdout:896: snap_create 2023-12-15T23:00:08.245 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T23:00:08.333 INFO:tasks.rados.rados.0.smithi071.stdout:896: done (0 left) 2023-12-15T23:00:08.333 INFO:tasks.rados.rados.0.smithi071.stdout:897: delete oid 13 current snap is 102 2023-12-15T23:00:08.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:08.359+0000 7f4322b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:08.364 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:08.363+0000 7f4322b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:08.369 INFO:tasks.rados.rados.0.smithi071.stdout:897: done (0 left) 2023-12-15T23:00:08.369 INFO:tasks.rados.rados.0.smithi071.stdout:898: delete oid 45 current snap is 102 2023-12-15T23:00:08.405 INFO:tasks.rados.rados.0.smithi071.stdout:898: done (0 left) 2023-12-15T23:00:08.406 INFO:tasks.rados.rados.0.smithi071.stdout:899: delete oid 48 current snap is 102 2023-12-15T23:00:08.466 INFO:tasks.rados.rados.0.smithi071.stdout:899: done (0 left) 2023-12-15T23:00:08.466 INFO:tasks.rados.rados.0.smithi071.stdout:900: read oid 11 snap -1 2023-12-15T23:00:08.466 INFO:tasks.rados.rados.0.smithi071.stdout:900: expect (ObjNum 390 snap 95 seq_num 390) 2023-12-15T23:00:08.466 INFO:tasks.rados.rados.0.smithi071.stdout:901: read oid 46 snap -1 2023-12-15T23:00:08.467 INFO:tasks.rados.rados.0.smithi071.stdout:901: expect deleted 2023-12-15T23:00:08.467 INFO:tasks.rados.rados.0.smithi071.stdout:902: read oid 16 snap -1 2023-12-15T23:00:08.467 INFO:tasks.rados.rados.0.smithi071.stdout:902: expect deleted 2023-12-15T23:00:08.467 INFO:tasks.rados.rados.0.smithi071.stdout:903: snap_create 2023-12-15T23:00:08.474 INFO:tasks.rados.rados.0.smithi071.stdout:900: expect (ObjNum 390 snap 95 seq_num 390) 2023-12-15T23:00:09.348 INFO:tasks.rados.rados.0.smithi071.stdout:900: done (3 left) 2023-12-15T23:00:09.348 INFO:tasks.rados.rados.0.smithi071.stdout:901: done (2 left) 2023-12-15T23:00:09.348 INFO:tasks.rados.rados.0.smithi071.stdout:902: done (1 left) 2023-12-15T23:00:09.348 INFO:tasks.rados.rados.0.smithi071.stdout:903: done (0 left) 2023-12-15T23:00:09.348 INFO:tasks.rados.rados.0.smithi071.stdout:904: delete oid 49 current snap is 103 2023-12-15T23:00:09.394 INFO:tasks.rados.rados.0.smithi071.stdout:904: done (0 left) 2023-12-15T23:00:09.394 INFO:tasks.rados.rados.0.smithi071.stdout:905: write oid 8 current snap is 103 2023-12-15T23:00:09.395 INFO:tasks.rados.rados.0.smithi071.stdout:905: seq_num 441 ranges {611887=733283,1972912=456484,2894128=517759} 2023-12-15T23:00:09.401 INFO:tasks.rados.rados.0.smithi071.stdout:905: writing smithi07183189-8 from 611887 to 1345170 tid 1 2023-12-15T23:00:09.407 INFO:tasks.rados.rados.0.smithi071.stdout:905: writing smithi07183189-8 from 1972912 to 2429396 tid 2 2023-12-15T23:00:09.413 INFO:tasks.rados.rados.0.smithi071.stdout:905: writing smithi07183189-8 from 2894128 to 3411887 tid 3 2023-12-15T23:00:09.413 INFO:tasks.rados.rados.0.smithi071.stdout:906: write (excl) oid 46 current snap is 103 2023-12-15T23:00:09.413 INFO:tasks.rados.rados.0.smithi071.stdout:906: seq_num 442 ranges {628694=408532,1515041=600142,2805198=552112} 2023-12-15T23:00:09.419 INFO:tasks.rados.rados.0.smithi071.stdout:906: writing smithi07183189-46 from 628694 to 1037226 tid 1 2023-12-15T23:00:09.425 INFO:tasks.rados.rados.0.smithi071.stdout:906: writing smithi07183189-46 from 1515041 to 2115183 tid 2 2023-12-15T23:00:09.432 INFO:tasks.rados.rados.0.smithi071.stdout:906: writing smithi07183189-46 from 2805198 to 3357310 tid 3 2023-12-15T23:00:09.432 INFO:tasks.rados.rados.0.smithi071.stdout:907: snap_remove snap 102 2023-12-15T23:00:10.349 INFO:tasks.rados.rados.0.smithi071.stdout:907: done (2 left) 2023-12-15T23:00:10.349 INFO:tasks.rados.rados.0.smithi071.stdout:908: snap_remove snap 101 2023-12-15T23:00:10.450 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T23:00:10.450 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2023-12-15T23:00:10.456 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T23:00:10.575 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:10.567+0000 7f4d39c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:10.578 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:10.578+0000 7f4d39c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:10.605 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:10.603+0000 7fb3ae22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:10.608 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:10.607+0000 7fb3ae22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:11.352 INFO:teuthology.orchestra.run.smithi071.stderr:reweighted osd.0 to 1 (10000) 2023-12-15T23:00:11.356 INFO:tasks.rados.rados.0.smithi071.stdout:908: done (2 left) 2023-12-15T23:00:11.356 INFO:tasks.rados.rados.0.smithi071.stdout:909: read oid 7 snap -1 2023-12-15T23:00:11.357 INFO:tasks.rados.rados.0.smithi071.stdout:909: expect (ObjNum 370 snap 91 seq_num 370) 2023-12-15T23:00:11.357 INFO:tasks.rados.rados.0.smithi071.stdout:910: snap_remove snap 100 2023-12-15T23:00:11.373 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2023-12-15T23:00:11.489 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:11.483+0000 7f505a66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:11.491 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:11.490+0000 7f505a66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:12.270 INFO:teuthology.orchestra.run.smithi071.stderr:reweighted osd.1 to 1 (10000) 2023-12-15T23:00:12.273 INFO:tasks.rados.rados.0.smithi071.stdout:910: done (3 left) 2023-12-15T23:00:12.273 INFO:tasks.rados.rados.0.smithi071.stdout:911: write oid 28 current snap is 103 2023-12-15T23:00:12.273 INFO:tasks.rados.rados.0.smithi071.stdout:911: seq_num 443 ranges {645501=483781,1573523=460153,2548974=763384} 2023-12-15T23:00:12.281 INFO:tasks.rados.rados.0.smithi071.stdout:911: writing smithi07183189-28 from 645501 to 1129282 tid 1 2023-12-15T23:00:12.285 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2023-12-15T23:00:12.288 INFO:tasks.rados.rados.0.smithi071.stdout:911: writing smithi07183189-28 from 1573523 to 2033676 tid 2 2023-12-15T23:00:12.298 INFO:tasks.rados.rados.0.smithi071.stdout:911: writing smithi07183189-28 from 2548974 to 3312358 tid 3 2023-12-15T23:00:12.298 INFO:tasks.rados.rados.0.smithi071.stdout:912: read oid 23 snap -1 2023-12-15T23:00:12.298 INFO:tasks.rados.rados.0.smithi071.stdout:912: expect deleted 2023-12-15T23:00:12.298 INFO:tasks.rados.rados.0.smithi071.stdout:913: delete oid 34 current snap is 103 2023-12-15T23:00:12.298 INFO:tasks.rados.rados.0.smithi071.stdout:906: finishing write tid 1 to smithi07183189-46 2023-12-15T23:00:12.298 INFO:tasks.rados.rados.0.smithi071.stdout:905: finishing write tid 1 to smithi07183189-8 2023-12-15T23:00:12.298 INFO:tasks.rados.rados.0.smithi071.stdout:906: finishing write tid 2 to smithi07183189-46 2023-12-15T23:00:12.298 INFO:tasks.rados.rados.0.smithi071.stdout:906: finishing write tid 3 to smithi07183189-46 2023-12-15T23:00:12.298 INFO:tasks.rados.rados.0.smithi071.stdout:905: finishing write tid 2 to smithi07183189-8 2023-12-15T23:00:12.298 INFO:tasks.rados.rados.0.smithi071.stdout:906: finishing write tid 4 to smithi07183189-46 2023-12-15T23:00:12.298 INFO:tasks.rados.rados.0.smithi071.stdout:906: finishing write tid 5 to smithi07183189-46 2023-12-15T23:00:12.298 INFO:tasks.rados.rados.0.smithi071.stdout:906: oid 46 updating version 0 to 138 2023-12-15T23:00:12.299 INFO:tasks.rados.rados.0.smithi071.stdout:906: oid 46 updating version 138 to 140 2023-12-15T23:00:12.299 INFO:tasks.rados.rados.0.smithi071.stdout:906: oid 46 updating version 140 to 141 2023-12-15T23:00:12.299 INFO:tasks.rados.rados.0.smithi071.stdout:906: oid 46 version 141 is already newer than 140 2023-12-15T23:00:12.299 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 141 (ObjNum 442 snap 103 seq_num 442) dirty exists 2023-12-15T23:00:12.299 INFO:tasks.rados.rados.0.smithi071.stdout:906: left oid 46 (ObjNum 442 snap 103 seq_num 442) 2023-12-15T23:00:12.299 INFO:tasks.rados.rados.0.smithi071.stdout:905: finishing write tid 3 to smithi07183189-8 2023-12-15T23:00:12.299 INFO:tasks.rados.rados.0.smithi071.stdout:905: finishing write tid 4 to smithi07183189-8 2023-12-15T23:00:12.299 INFO:tasks.rados.rados.0.smithi071.stdout:905: finishing write tid 5 to smithi07183189-8 2023-12-15T23:00:12.299 INFO:tasks.rados.rados.0.smithi071.stdout:905: oid 8 updating version 0 to 265 2023-12-15T23:00:12.299 INFO:tasks.rados.rados.0.smithi071.stdout:905: oid 8 updating version 265 to 268 2023-12-15T23:00:12.299 INFO:tasks.rados.rados.0.smithi071.stdout:905: oid 8 version 268 is already newer than 268 2023-12-15T23:00:12.299 INFO:tasks.rados.rados.0.smithi071.stdout:905: oid 8 version 268 is already newer than 266 2023-12-15T23:00:12.299 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 8 v 268 (ObjNum 441 snap 103 seq_num 441) dirty exists 2023-12-15T23:00:12.299 INFO:tasks.rados.rados.0.smithi071.stdout:905: left oid 8 (ObjNum 441 snap 103 seq_num 441) 2023-12-15T23:00:12.299 INFO:tasks.rados.rados.0.smithi071.stdout:909: expect (ObjNum 370 snap 91 seq_num 370) 2023-12-15T23:00:12.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:12.400+0000 7fbe1059e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:12.407 INFO:tasks.rados.rados.0.smithi071.stdout:905: done (5 left) 2023-12-15T23:00:12.407 INFO:tasks.rados.rados.0.smithi071.stdout:906: done (4 left) 2023-12-15T23:00:12.407 INFO:tasks.rados.rados.0.smithi071.stdout:909: done (3 left) 2023-12-15T23:00:12.407 INFO:tasks.rados.rados.0.smithi071.stdout:912: done (2 left) 2023-12-15T23:00:12.407 INFO:tasks.rados.rados.0.smithi071.stdout:913: done (1 left) 2023-12-15T23:00:12.407 INFO:tasks.rados.rados.0.smithi071.stdout:914: read oid 10 snap -1 2023-12-15T23:00:12.407 INFO:tasks.rados.rados.0.smithi071.stdout:914: expect deleted 2023-12-15T23:00:12.407 INFO:tasks.rados.rados.0.smithi071.stdout:915: write oid 24 current snap is 103 2023-12-15T23:00:12.407 INFO:tasks.rados.rados.0.smithi071.stdout:915: seq_num 445 ranges {679115=634279,2090487=463822,3003820=475295} 2023-12-15T23:00:12.409 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:12.409+0000 7fbe1059e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:12.415 INFO:tasks.rados.rados.0.smithi071.stdout:915: writing smithi07183189-24 from 679115 to 1313394 tid 1 2023-12-15T23:00:12.422 INFO:tasks.rados.rados.0.smithi071.stdout:915: writing smithi07183189-24 from 2090487 to 2554309 tid 2 2023-12-15T23:00:12.428 INFO:tasks.rados.rados.0.smithi071.stdout:915: writing smithi07183189-24 from 3003820 to 3479115 tid 3 2023-12-15T23:00:12.428 INFO:tasks.rados.rados.0.smithi071.stdout:916: read oid 18 snap -1 2023-12-15T23:00:12.428 INFO:tasks.rados.rados.0.smithi071.stdout:916: expect deleted 2023-12-15T23:00:12.428 INFO:tasks.rados.rados.0.smithi071.stdout:917: delete oid 35 current snap is 103 2023-12-15T23:00:12.428 INFO:tasks.rados.rados.0.smithi071.stdout:911: finishing write tid 1 to smithi07183189-28 2023-12-15T23:00:12.429 INFO:tasks.rados.rados.0.smithi071.stdout:911: finishing write tid 2 to smithi07183189-28 2023-12-15T23:00:12.462 INFO:tasks.rados.rados.0.smithi071.stdout:911: finishing write tid 3 to smithi07183189-28 2023-12-15T23:00:12.471 INFO:tasks.rados.rados.0.smithi071.stdout:916: done (4 left) 2023-12-15T23:00:12.471 INFO:tasks.rados.rados.0.smithi071.stdout:917: done (3 left) 2023-12-15T23:00:12.471 INFO:tasks.rados.rados.0.smithi071.stdout:918: read oid 27 snap -1 2023-12-15T23:00:12.471 INFO:tasks.rados.rados.0.smithi071.stdout:918: expect (ObjNum 420 snap 99 seq_num 420) 2023-12-15T23:00:12.471 INFO:tasks.rados.rados.0.smithi071.stdout:919: delete oid 26 current snap is 103 2023-12-15T23:00:12.471 INFO:tasks.rados.rados.0.smithi071.stdout:911: finishing write tid 4 to smithi07183189-28 2023-12-15T23:00:12.481 INFO:tasks.rados.rados.0.smithi071.stdout:911: finishing write tid 5 to smithi07183189-28 2023-12-15T23:00:12.481 INFO:tasks.rados.rados.0.smithi071.stdout:911: oid 28 updating version 0 to 158 2023-12-15T23:00:12.481 INFO:tasks.rados.rados.0.smithi071.stdout:911: oid 28 version 158 is already newer than 156 2023-12-15T23:00:12.481 INFO:tasks.rados.rados.0.smithi071.stdout:911: oid 28 version 158 is already newer than 157 2023-12-15T23:00:12.481 INFO:tasks.rados.rados.0.smithi071.stdout:911: oid 28 updating version 158 to 159 2023-12-15T23:00:12.481 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 28 v 159 (ObjNum 443 snap 103 seq_num 443) dirty exists 2023-12-15T23:00:12.481 INFO:tasks.rados.rados.0.smithi071.stdout:911: left oid 28 (ObjNum 443 snap 103 seq_num 443) 2023-12-15T23:00:12.497 INFO:tasks.rados.rados.0.smithi071.stdout:911: done (4 left) 2023-12-15T23:00:12.497 INFO:tasks.rados.rados.0.smithi071.stdout:914: done (3 left) 2023-12-15T23:00:12.497 INFO:tasks.rados.rados.0.smithi071.stdout:919: done (2 left) 2023-12-15T23:00:12.497 INFO:tasks.rados.rados.0.smithi071.stdout:920: write (excl) oid 30 current snap is 103 2023-12-15T23:00:12.497 INFO:tasks.rados.rados.0.smithi071.stdout:920: seq_num 448 ranges {729536=460026,1749580=611149,2853383=452450} 2023-12-15T23:00:12.504 INFO:tasks.rados.rados.0.smithi071.stdout:920: writing smithi07183189-30 from 729536 to 1189562 tid 1 2023-12-15T23:00:12.512 INFO:tasks.rados.rados.0.smithi071.stdout:920: writing smithi07183189-30 from 1749580 to 2360729 tid 2 2023-12-15T23:00:12.518 INFO:tasks.rados.rados.0.smithi071.stdout:920: writing smithi07183189-30 from 2853383 to 3305833 tid 3 2023-12-15T23:00:12.518 INFO:tasks.rados.rados.0.smithi071.stdout:921: write (excl) oid 40 current snap is 103 2023-12-15T23:00:12.518 INFO:tasks.rados.rados.0.smithi071.stdout:921: seq_num 449 ranges {746343=651628,1924415=754807,3397159=149184} 2023-12-15T23:00:12.528 INFO:tasks.rados.rados.0.smithi071.stdout:921: writing smithi07183189-40 from 746343 to 1397971 tid 1 2023-12-15T23:00:12.537 INFO:tasks.rados.rados.0.smithi071.stdout:921: writing smithi07183189-40 from 1924415 to 2679222 tid 2 2023-12-15T23:00:12.542 INFO:tasks.rados.rados.0.smithi071.stdout:921: writing smithi07183189-40 from 3397159 to 3546343 tid 3 2023-12-15T23:00:12.543 INFO:tasks.rados.rados.0.smithi071.stdout:922: snap_create 2023-12-15T23:00:12.543 INFO:tasks.rados.rados.0.smithi071.stdout:915: finishing write tid 1 to smithi07183189-24 2023-12-15T23:00:12.543 INFO:tasks.rados.rados.0.smithi071.stdout:918: expect (ObjNum 420 snap 99 seq_num 420) 2023-12-15T23:00:12.674 INFO:tasks.rados.rados.0.smithi071.stdout:920: finishing write tid 1 to smithi07183189-30 2023-12-15T23:00:12.674 INFO:tasks.rados.rados.0.smithi071.stdout:915: finishing write tid 2 to smithi07183189-24 2023-12-15T23:00:12.675 INFO:tasks.rados.rados.0.smithi071.stdout:920: finishing write tid 2 to smithi07183189-30 2023-12-15T23:00:12.675 INFO:tasks.rados.rados.0.smithi071.stdout:921: finishing write tid 1 to smithi07183189-40 2023-12-15T23:00:12.675 INFO:tasks.rados.rados.0.smithi071.stdout:915: finishing write tid 3 to smithi07183189-24 2023-12-15T23:00:12.675 INFO:tasks.rados.rados.0.smithi071.stdout:915: finishing write tid 4 to smithi07183189-24 2023-12-15T23:00:12.675 INFO:tasks.rados.rados.0.smithi071.stdout:920: finishing write tid 3 to smithi07183189-30 2023-12-15T23:00:12.675 INFO:tasks.rados.rados.0.smithi071.stdout:915: finishing write tid 5 to smithi07183189-24 2023-12-15T23:00:12.675 INFO:tasks.rados.rados.0.smithi071.stdout:915: oid 24 updating version 0 to 103 2023-12-15T23:00:12.675 INFO:tasks.rados.rados.0.smithi071.stdout:915: oid 24 version 103 is already newer than 101 2023-12-15T23:00:12.675 INFO:tasks.rados.rados.0.smithi071.stdout:915: oid 24 version 103 is already newer than 103 2023-12-15T23:00:12.675 INFO:tasks.rados.rados.0.smithi071.stdout:915: oid 24 version 103 is already newer than 100 2023-12-15T23:00:12.675 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 103 (ObjNum 445 snap 103 seq_num 445) dirty exists 2023-12-15T23:00:12.675 INFO:tasks.rados.rados.0.smithi071.stdout:915: left oid 24 (ObjNum 445 snap 103 seq_num 445) 2023-12-15T23:00:12.675 INFO:tasks.rados.rados.0.smithi071.stdout:920: finishing write tid 4 to smithi07183189-30 2023-12-15T23:00:12.675 INFO:tasks.rados.rados.0.smithi071.stdout:920: finishing write tid 5 to smithi07183189-30 2023-12-15T23:00:12.676 INFO:tasks.rados.rados.0.smithi071.stdout:920: oid 30 updating version 0 to 123 2023-12-15T23:00:12.676 INFO:tasks.rados.rados.0.smithi071.stdout:920: oid 30 version 123 is already newer than 121 2023-12-15T23:00:12.676 INFO:tasks.rados.rados.0.smithi071.stdout:920: oid 30 version 123 is already newer than 122 2023-12-15T23:00:12.676 INFO:tasks.rados.rados.0.smithi071.stdout:920: oid 30 updating version 123 to 124 2023-12-15T23:00:12.676 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 124 (ObjNum 448 snap 103 seq_num 448) dirty exists 2023-12-15T23:00:12.676 INFO:tasks.rados.rados.0.smithi071.stdout:920: left oid 30 (ObjNum 448 snap 103 seq_num 448) 2023-12-15T23:00:12.676 INFO:tasks.rados.rados.0.smithi071.stdout:921: finishing write tid 2 to smithi07183189-40 2023-12-15T23:00:12.676 INFO:tasks.rados.rados.0.smithi071.stdout:921: finishing write tid 3 to smithi07183189-40 2023-12-15T23:00:12.676 INFO:tasks.rados.rados.0.smithi071.stdout:921: finishing write tid 4 to smithi07183189-40 2023-12-15T23:00:12.676 INFO:tasks.rados.rados.0.smithi071.stdout:921: finishing write tid 5 to smithi07183189-40 2023-12-15T23:00:12.676 INFO:tasks.rados.rados.0.smithi071.stdout:921: oid 40 updating version 0 to 582 2023-12-15T23:00:12.676 INFO:tasks.rados.rados.0.smithi071.stdout:921: oid 40 updating version 582 to 583 2023-12-15T23:00:12.676 INFO:tasks.rados.rados.0.smithi071.stdout:921: oid 40 version 583 is already newer than 582 2023-12-15T23:00:12.676 INFO:tasks.rados.rados.0.smithi071.stdout:921: oid 40 version 583 is already newer than 580 2023-12-15T23:00:12.677 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 40 v 583 (ObjNum 449 snap 103 seq_num 449) dirty exists 2023-12-15T23:00:12.677 INFO:tasks.rados.rados.0.smithi071.stdout:921: left oid 40 (ObjNum 449 snap 103 seq_num 449) 2023-12-15T23:00:12.677 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T23:00:12.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:12.778+0000 7f898659e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:12.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:12.782+0000 7f898659e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:13.357 INFO:teuthology.orchestra.run.smithi071.stderr:reweighted osd.2 to 1 (10000) 2023-12-15T23:00:13.360 INFO:tasks.rados.rados.0.smithi071.stdout:915: done (4 left) 2023-12-15T23:00:13.360 INFO:tasks.rados.rados.0.smithi071.stdout:918: done (3 left) 2023-12-15T23:00:13.360 INFO:tasks.rados.rados.0.smithi071.stdout:920: done (2 left) 2023-12-15T23:00:13.360 INFO:tasks.rados.rados.0.smithi071.stdout:921: done (1 left) 2023-12-15T23:00:13.361 INFO:tasks.rados.rados.0.smithi071.stdout:922: done (0 left) 2023-12-15T23:00:13.361 INFO:tasks.rados.rados.0.smithi071.stdout:923: read oid 48 snap -1 2023-12-15T23:00:13.361 INFO:tasks.rados.rados.0.smithi071.stdout:923: expect deleted 2023-12-15T23:00:13.361 INFO:tasks.rados.rados.0.smithi071.stdout:924: delete oid 38 current snap is 104 2023-12-15T23:00:13.373 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2023-12-15T23:00:13.417 INFO:tasks.rados.rados.0.smithi071.stdout:923: done (1 left) 2023-12-15T23:00:13.417 INFO:tasks.rados.rados.0.smithi071.stdout:924: done (0 left) 2023-12-15T23:00:13.417 INFO:tasks.rados.rados.0.smithi071.stdout:925: write oid 33 current snap is 104 2023-12-15T23:00:13.417 INFO:tasks.rados.rados.0.smithi071.stdout:925: seq_num 451 ranges {779957=402126,1925026=758476,3335652=244305} 2023-12-15T23:00:13.423 INFO:tasks.rados.rados.0.smithi071.stdout:925: writing smithi07183189-33 from 779957 to 1182083 tid 1 2023-12-15T23:00:13.434 INFO:tasks.rados.rados.0.smithi071.stdout:925: writing smithi07183189-33 from 1925026 to 2683502 tid 2 2023-12-15T23:00:13.439 INFO:tasks.rados.rados.0.smithi071.stdout:925: writing smithi07183189-33 from 3335652 to 3579957 tid 3 2023-12-15T23:00:13.440 INFO:tasks.rados.rados.0.smithi071.stdout:926: write oid 26 current snap is 104 2023-12-15T23:00:13.440 INFO:tasks.rados.rados.0.smithi071.stdout:926: seq_num 452 ranges {796764=477375,1983508=618487,3363075=233689} 2023-12-15T23:00:13.448 INFO:tasks.rados.rados.0.smithi071.stdout:926: writing smithi07183189-26 from 796764 to 1274139 tid 1 2023-12-15T23:00:13.458 INFO:tasks.rados.rados.0.smithi071.stdout:926: writing smithi07183189-26 from 1983508 to 2601995 tid 2 2023-12-15T23:00:13.464 INFO:tasks.rados.rados.0.smithi071.stdout:926: writing smithi07183189-26 from 3363075 to 3596764 tid 3 2023-12-15T23:00:13.464 INFO:tasks.rados.rados.0.smithi071.stdout:927: write (excl) oid 27 current snap is 104 2023-12-15T23:00:13.464 INFO:tasks.rados.rados.0.smithi071.stdout:925: finishing write tid 1 to smithi07183189-33 2023-12-15T23:00:13.464 INFO:tasks.rados.rados.0.smithi071.stdout:927: seq_num 453 ranges {413571=552624,1641990=762145,2990498=425163} 2023-12-15T23:00:13.471 INFO:tasks.rados.rados.0.smithi071.stdout:927: writing smithi07183189-27 from 413571 to 966195 tid 1 2023-12-15T23:00:13.482 INFO:tasks.rados.rados.0.smithi071.stdout:927: writing smithi07183189-27 from 1641990 to 2404135 tid 2 2023-12-15T23:00:13.489 INFO:tasks.rados.rados.0.smithi071.stdout:927: writing smithi07183189-27 from 2990498 to 3415661 tid 3 2023-12-15T23:00:13.489 INFO:tasks.rados.rados.0.smithi071.stdout:928: write (excl) oid 19 current snap is 104 2023-12-15T23:00:13.489 INFO:tasks.rados.rados.0.smithi071.stdout:925: finishing write tid 2 to smithi07183189-33 2023-12-15T23:00:13.489 INFO:tasks.rados.rados.0.smithi071.stdout:928: seq_num 454 ranges {430378=627873,1700472=622156,3017921=612457} 2023-12-15T23:00:13.492 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:13.485+0000 7f30ab71e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:13.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:13.496+0000 7f30ab71e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:13.497 INFO:tasks.rados.rados.0.smithi071.stdout:928: writing smithi07183189-19 from 430378 to 1058251 tid 1 2023-12-15T23:00:13.505 INFO:tasks.rados.rados.0.smithi071.stdout:928: writing smithi07183189-19 from 1700472 to 2322628 tid 2 2023-12-15T23:00:13.515 INFO:tasks.rados.rados.0.smithi071.stdout:928: writing smithi07183189-19 from 3017921 to 3630378 tid 3 2023-12-15T23:00:13.515 INFO:tasks.rados.rados.0.smithi071.stdout:929: delete oid 21 current snap is 104 2023-12-15T23:00:13.515 INFO:tasks.rados.rados.0.smithi071.stdout:926: finishing write tid 1 to smithi07183189-26 2023-12-15T23:00:13.538 INFO:tasks.rados.rados.0.smithi071.stdout:925: finishing write tid 3 to smithi07183189-33 2023-12-15T23:00:13.570 INFO:tasks.rados.rados.0.smithi071.stdout:926: finishing write tid 2 to smithi07183189-26 2023-12-15T23:00:13.571 INFO:tasks.rados.rados.0.smithi071.stdout:925: finishing write tid 4 to smithi07183189-33 2023-12-15T23:00:13.573 INFO:tasks.rados.rados.0.smithi071.stdout:929: done (4 left) 2023-12-15T23:00:13.573 INFO:tasks.rados.rados.0.smithi071.stdout:930: delete oid 12 current snap is 104 2023-12-15T23:00:13.575 INFO:tasks.rados.rados.0.smithi071.stdout:925: finishing write tid 5 to smithi07183189-33 2023-12-15T23:00:13.575 INFO:tasks.rados.rados.0.smithi071.stdout:925: oid 33 updating version 0 to 588 2023-12-15T23:00:13.575 INFO:tasks.rados.rados.0.smithi071.stdout:925: oid 33 version 588 is already newer than 585 2023-12-15T23:00:13.575 INFO:tasks.rados.rados.0.smithi071.stdout:925: oid 33 version 588 is already newer than 586 2023-12-15T23:00:13.575 INFO:tasks.rados.rados.0.smithi071.stdout:925: oid 33 version 588 is already newer than 587 2023-12-15T23:00:13.575 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 33 v 588 (ObjNum 451 snap 104 seq_num 451) dirty exists 2023-12-15T23:00:13.575 INFO:tasks.rados.rados.0.smithi071.stdout:925: left oid 33 (ObjNum 451 snap 104 seq_num 451) 2023-12-15T23:00:13.577 INFO:tasks.rados.rados.0.smithi071.stdout:927: finishing write tid 1 to smithi07183189-27 2023-12-15T23:00:13.579 INFO:tasks.rados.rados.0.smithi071.stdout:928: finishing write tid 1 to smithi07183189-19 2023-12-15T23:00:13.596 INFO:tasks.rados.rados.0.smithi071.stdout:926: finishing write tid 3 to smithi07183189-26 2023-12-15T23:00:13.614 INFO:tasks.rados.rados.0.smithi071.stdout:926: finishing write tid 4 to smithi07183189-26 2023-12-15T23:00:13.615 INFO:tasks.rados.rados.0.smithi071.stdout:926: finishing write tid 5 to smithi07183189-26 2023-12-15T23:00:13.615 INFO:tasks.rados.rados.0.smithi071.stdout:926: oid 26 updating version 0 to 192 2023-12-15T23:00:13.615 INFO:tasks.rados.rados.0.smithi071.stdout:926: oid 26 updating version 192 to 193 2023-12-15T23:00:13.615 INFO:tasks.rados.rados.0.smithi071.stdout:926: oid 26 updating version 193 to 195 2023-12-15T23:00:13.615 INFO:tasks.rados.rados.0.smithi071.stdout:926: oid 26 version 195 is already newer than 195 2023-12-15T23:00:13.615 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 26 v 195 (ObjNum 452 snap 104 seq_num 452) dirty exists 2023-12-15T23:00:13.615 INFO:tasks.rados.rados.0.smithi071.stdout:926: left oid 26 (ObjNum 452 snap 104 seq_num 452) 2023-12-15T23:00:13.616 INFO:tasks.rados.rados.0.smithi071.stdout:927: finishing write tid 2 to smithi07183189-27 2023-12-15T23:00:13.625 INFO:tasks.rados.rados.0.smithi071.stdout:928: finishing write tid 2 to smithi07183189-19 2023-12-15T23:00:13.633 INFO:tasks.rados.rados.0.smithi071.stdout:925: done (4 left) 2023-12-15T23:00:13.633 INFO:tasks.rados.rados.0.smithi071.stdout:926: done (3 left) 2023-12-15T23:00:13.633 INFO:tasks.rados.rados.0.smithi071.stdout:930: done (2 left) 2023-12-15T23:00:13.633 INFO:tasks.rados.rados.0.smithi071.stdout:931: snap_create 2023-12-15T23:00:13.636 INFO:tasks.rados.rados.0.smithi071.stdout:928: finishing write tid 3 to smithi07183189-19 2023-12-15T23:00:13.642 INFO:tasks.rados.rados.0.smithi071.stdout:927: finishing write tid 3 to smithi07183189-27 2023-12-15T23:00:13.650 INFO:tasks.rados.rados.0.smithi071.stdout:928: finishing write tid 4 to smithi07183189-19 2023-12-15T23:00:13.652 INFO:tasks.rados.rados.0.smithi071.stdout:928: finishing write tid 5 to smithi07183189-19 2023-12-15T23:00:13.652 INFO:tasks.rados.rados.0.smithi071.stdout:928: oid 19 updating version 0 to 296 2023-12-15T23:00:13.652 INFO:tasks.rados.rados.0.smithi071.stdout:928: oid 19 version 296 is already newer than 294 2023-12-15T23:00:13.652 INFO:tasks.rados.rados.0.smithi071.stdout:928: oid 19 version 296 is already newer than 296 2023-12-15T23:00:13.652 INFO:tasks.rados.rados.0.smithi071.stdout:928: oid 19 updating version 296 to 297 2023-12-15T23:00:13.652 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 297 (ObjNum 454 snap 104 seq_num 454) dirty exists 2023-12-15T23:00:13.652 INFO:tasks.rados.rados.0.smithi071.stdout:928: left oid 19 (ObjNum 454 snap 104 seq_num 454) 2023-12-15T23:00:13.653 INFO:tasks.rados.rados.0.smithi071.stdout:927: finishing write tid 4 to smithi07183189-27 2023-12-15T23:00:13.654 INFO:tasks.rados.rados.0.smithi071.stdout:927: finishing write tid 5 to smithi07183189-27 2023-12-15T23:00:13.654 INFO:tasks.rados.rados.0.smithi071.stdout:927: oid 27 updating version 0 to 224 2023-12-15T23:00:13.654 INFO:tasks.rados.rados.0.smithi071.stdout:927: oid 27 updating version 224 to 226 2023-12-15T23:00:13.655 INFO:tasks.rados.rados.0.smithi071.stdout:927: oid 27 version 226 is already newer than 223 2023-12-15T23:00:13.655 INFO:tasks.rados.rados.0.smithi071.stdout:927: oid 27 version 226 is already newer than 225 2023-12-15T23:00:13.655 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 226 (ObjNum 453 snap 104 seq_num 453) dirty exists 2023-12-15T23:00:13.655 INFO:tasks.rados.rados.0.smithi071.stdout:927: left oid 27 (ObjNum 453 snap 104 seq_num 453) 2023-12-15T23:00:14.359 INFO:teuthology.orchestra.run.smithi071.stderr:reweighted osd.3 to 1 (10000) 2023-12-15T23:00:14.363 INFO:tasks.rados.rados.0.smithi071.stdout:927: done (2 left) 2023-12-15T23:00:14.363 INFO:tasks.rados.rados.0.smithi071.stdout:928: done (1 left) 2023-12-15T23:00:14.363 INFO:tasks.rados.rados.0.smithi071.stdout:931: done (0 left) 2023-12-15T23:00:14.363 INFO:tasks.rados.rados.0.smithi071.stdout:932: read oid 42 snap -1 2023-12-15T23:00:14.363 INFO:tasks.rados.rados.0.smithi071.stdout:932: expect (ObjNum 428 snap 100 seq_num 428) 2023-12-15T23:00:14.363 INFO:tasks.rados.rados.0.smithi071.stdout:933: read oid 13 snap 103 2023-12-15T23:00:14.363 INFO:tasks.rados.rados.0.smithi071.stdout:933: expect deleted 2023-12-15T23:00:14.363 INFO:tasks.rados.rados.0.smithi071.stdout:934: read oid 41 snap 104 2023-12-15T23:00:14.363 INFO:tasks.rados.rados.0.smithi071.stdout:934: expect (ObjNum 365 snap 91 seq_num 365) 2023-12-15T23:00:14.364 INFO:tasks.rados.rados.0.smithi071.stdout:935: write (excl) oid 48 current snap is 105 2023-12-15T23:00:14.364 INFO:tasks.rados.rados.0.smithi071.stdout:935: seq_num 457 ranges {480799=569973,1475918=769483,2700190=586604} 2023-12-15T23:00:14.368 INFO:tasks.rados.rados.0.smithi071.stdout:935: writing smithi07183189-48 from 480799 to 1050772 tid 1 2023-12-15T23:00:14.375 INFO:tasks.rados.rados.0.smithi071.stdout:935: writing smithi07183189-48 from 1475918 to 2245401 tid 2 2023-12-15T23:00:14.376 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2023-12-15T23:00:14.376 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T23:00:14.383 INFO:tasks.rados.rados.0.smithi071.stdout:935: writing smithi07183189-48 from 2700190 to 3286794 tid 3 2023-12-15T23:00:14.383 INFO:tasks.rados.rados.0.smithi071.stdout:936: write (excl) oid 8 current snap is 105 2023-12-15T23:00:14.383 INFO:tasks.rados.rados.0.smithi071.stdout:936: seq_num 458 ranges {497606=645222,1934400=629494,3243966=453640} 2023-12-15T23:00:14.391 INFO:tasks.rados.rados.0.smithi071.stdout:936: writing smithi07183189-8 from 497606 to 1142828 tid 1 2023-12-15T23:00:14.399 INFO:tasks.rados.rados.0.smithi071.stdout:936: writing smithi07183189-8 from 1934400 to 2563894 tid 2 2023-12-15T23:00:14.406 INFO:tasks.rados.rados.0.smithi071.stdout:936: writing smithi07183189-8 from 3243966 to 3697606 tid 3 2023-12-15T23:00:14.406 INFO:tasks.rados.rados.0.smithi071.stdout:937: snap_create 2023-12-15T23:00:14.406 INFO:tasks.rados.rados.0.smithi071.stdout:934: expect (ObjNum 365 snap 91 seq_num 365) 2023-12-15T23:00:14.494 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:14.491+0000 7f35cce0f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:14.499 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:14.499+0000 7f35cce0f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:14.594 INFO:tasks.rados.rados.0.smithi071.stdout:932: expect (ObjNum 428 snap 100 seq_num 428) 2023-12-15T23:00:14.681 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T23:00:14.682 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-12-15T23:00:14.693 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":332,"stamp":"2023-12-15T23:00:14.554580+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98358436,"num_objects":300,"num_object_clones":214,"num_object_copies":900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":52,"num_read":1805,"num_read_kb":1260655,"num_write":1516,"num_write_kb":403510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3125,"ondisk_log_size":3125,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":326008,"kb_used_data":193752,"kb_used_omap":0,"kb_used_meta":132224,"kb_avail":377161352,"statfs":{"total":386547056640,"available":386213224448,"internally_reserved":0,"allocated":198402048,"data_stored":197564989,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":135397376},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2756348,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":17,"num_read_kb":19314,"num_write":20,"num_write_kb":6348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000589"},"pg_stats":[{"pgid":"3.f","version":"214'96","reported_seq":524,"reported_epoch":222,"state":"active+clean","last_fresh":"2023-12-15T23:00:10.540610+0000","last_change":"2023-12-15T23:00:10.558697+0000","last_active":"2023-12-15T23:00:10.540610+0000","last_peered":"2023-12-15T23:00:10.540610+0000","last_clean":"2023-12-15T23:00:10.540610+0000","last_became_active":"2023-12-15T22:56:39.503797+0000","last_became_peered":"2023-12-15T22:56:39.503797+0000","last_unstale":"2023-12-15T23:00:10.540610+0000","last_undegraded":"2023-12-15T23:00:10.540610+0000","last_fullsized":"2023-12-15T23:00:10.540610+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6084233,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":78,"num_read_kb":60669,"num_write":40,"num_write_kb":11941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":423,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-12-15T23:00:09.365470+0000","last_change":"2023-12-15T23:00:06.357073+0000","last_active":"2023-12-15T23:00:09.365470+0000","last_peered":"2023-12-15T23:00:09.365470+0000","last_clean":"2023-12-15T23:00:09.365470+0000","last_became_active":"2023-12-15T22:56:39.552856+0000","last_became_peered":"2023-12-15T22:56:39.552856+0000","last_unstale":"2023-12-15T23:00:09.365470+0000","last_undegraded":"2023-12-15T23:00:09.365470+0000","last_fullsized":"2023-12-15T23:00:09.365470+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"218'311","reported_seq":684,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-12-15T23:00:09.361441+0000","last_change":"2023-12-15T23:00:06.509587+0000","last_active":"2023-12-15T23:00:09.361441+0000","last_peered":"2023-12-15T23:00:09.361441+0000","last_clean":"2023-12-15T23:00:09.361441+0000","last_became_active":"2023-12-15T22:56:39.495490+0000","last_became_peered":"2023-12-15T22:56:39.495490+0000","last_unstale":"2023-12-15T23:00:09.361441+0000","last_undegraded":"2023-12-15T23:00:09.361441+0000","last_fullsized":"2023-12-15T23:00:09.361441+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14941687,"num_objects":33,"num_object_clones":22,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":231,"num_read_kb":202767,"num_write":169,"num_write_kb":48561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"208'291","reported_seq":653,"reported_epoch":222,"state":"active+clean","last_fresh":"2023-12-15T23:00:10.539998+0000","last_change":"2023-12-15T23:00:10.560071+0000","last_active":"2023-12-15T23:00:10.539998+0000","last_peered":"2023-12-15T23:00:10.539998+0000","last_clean":"2023-12-15T23:00:10.539998+0000","last_became_active":"2023-12-15T22:56:39.579234+0000","last_became_peered":"2023-12-15T22:56:39.579234+0000","last_unstale":"2023-12-15T23:00:10.539998+0000","last_undegraded":"2023-12-15T23:00:10.539998+0000","last_fullsized":"2023-12-15T23:00:10.539998+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8891068,"num_objects":28,"num_object_clones":24,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":155,"num_read_kb":112879,"num_write":147,"num_write_kb":42462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"215'155","reported_seq":529,"reported_epoch":222,"state":"active+clean+snaptrim","last_fresh":"2023-12-15T23:00:10.573319+0000","last_change":"2023-12-15T23:00:10.573319+0000","last_active":"2023-12-15T23:00:10.573319+0000","last_peered":"2023-12-15T23:00:10.573319+0000","last_clean":"2023-12-15T23:00:10.573319+0000","last_became_active":"2023-12-15T22:56:39.572176+0000","last_became_peered":"2023-12-15T22:56:39.572176+0000","last_unstale":"2023-12-15T23:00:10.573319+0000","last_undegraded":"2023-12-15T23:00:10.573319+0000","last_fullsized":"2023-12-15T23:00:10.573319+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7008618,"num_objects":15,"num_object_clones":10,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":66,"num_read_kb":38159,"num_write":65,"num_write_kb":19531,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"218'145","reported_seq":526,"reported_epoch":222,"state":"active+clean","last_fresh":"2023-12-15T23:00:10.555322+0000","last_change":"2023-12-15T23:00:10.559235+0000","last_active":"2023-12-15T23:00:10.555322+0000","last_peered":"2023-12-15T23:00:10.555322+0000","last_clean":"2023-12-15T23:00:10.555322+0000","last_became_active":"2023-12-15T22:56:39.516361+0000","last_became_peered":"2023-12-15T22:56:39.516361+0000","last_unstale":"2023-12-15T23:00:10.555322+0000","last_undegraded":"2023-12-15T23:00:10.555322+0000","last_fullsized":"2023-12-15T23:00:10.555322+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3327852,"num_objects":13,"num_object_clones":9,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":52,"num_read_kb":38718,"num_write":56,"num_write_kb":16218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"220'577","reported_seq":781,"reported_epoch":222,"state":"active+clean","last_fresh":"2023-12-15T23:00:10.552582+0000","last_change":"2023-12-15T23:00:10.558974+0000","last_active":"2023-12-15T23:00:10.552582+0000","last_peered":"2023-12-15T23:00:10.552582+0000","last_clean":"2023-12-15T23:00:10.552582+0000","last_became_active":"2023-12-15T22:56:39.519748+0000","last_became_peered":"2023-12-15T22:56:39.519748+0000","last_unstale":"2023-12-15T23:00:10.552582+0000","last_undegraded":"2023-12-15T23:00:10.552582+0000","last_fullsized":"2023-12-15T23:00:10.552582+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":577,"log_dups_size":0,"ondisk_log_size":577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7193495,"num_objects":53,"num_object_clones":43,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":246,"num_read_kb":159819,"num_write":232,"num_write_kb":66545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"215'137","reported_seq":545,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-12-15T23:00:09.360192+0000","last_change":"2023-12-15T23:00:06.357099+0000","last_active":"2023-12-15T23:00:09.360192+0000","last_peered":"2023-12-15T23:00:09.360192+0000","last_clean":"2023-12-15T23:00:09.360192+0000","last_became_active":"2023-12-15T22:56:39.543179+0000","last_became_peered":"2023-12-15T22:56:39.543179+0000","last_unstale":"2023-12-15T23:00:09.360192+0000","last_undegraded":"2023-12-15T23:00:09.360192+0000","last_fullsized":"2023-12-15T23:00:09.360192+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8336371,"num_objects":13,"num_object_clones":9,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":123,"num_read_kb":109406,"num_write":74,"num_write_kb":19954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":468,"reported_epoch":222,"state":"active+clean","last_fresh":"2023-12-15T23:00:09.418014+0000","last_change":"2023-12-15T22:56:32.607017+0000","last_active":"2023-12-15T23:00:09.418014+0000","last_peered":"2023-12-15T23:00:09.418014+0000","last_clean":"2023-12-15T23:00:09.418014+0000","last_became_active":"2023-12-15T22:56:32.602384+0000","last_became_peered":"2023-12-15T22:56:32.602384+0000","last_unstale":"2023-12-15T23:00:09.418014+0000","last_undegraded":"2023-12-15T23:00:09.418014+0000","last_fullsized":"2023-12-15T23:00:09.418014+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_clean_scrub_stamp":"2023-12-15T22:56:31.525572+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":327,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-12-15T23:00:09.362971+0000","last_change":"2023-12-15T22:56:33.627481+0000","last_active":"2023-12-15T23:00:09.362971+0000","last_peered":"2023-12-15T23:00:09.362971+0000","last_clean":"2023-12-15T23:00:09.362971+0000","last_became_active":"2023-12-15T22:56:33.624689+0000","last_became_peered":"2023-12-15T22:56:33.624689+0000","last_unstale":"2023-12-15T23:00:09.362971+0000","last_undegraded":"2023-12-15T23:00:09.362971+0000","last_fullsized":"2023-12-15T23:00:09.362971+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"220'211","reported_seq":617,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-12-15T23:00:09.357171+0000","last_change":"2023-12-15T23:00:06.386625+0000","last_active":"2023-12-15T23:00:09.357171+0000","last_peered":"2023-12-15T23:00:09.357171+0000","last_clean":"2023-12-15T23:00:09.357171+0000","last_became_active":"2023-12-15T22:56:39.529359+0000","last_became_peered":"2023-12-15T22:56:39.529359+0000","last_unstale":"2023-12-15T23:00:09.357171+0000","last_undegraded":"2023-12-15T23:00:09.357171+0000","last_fullsized":"2023-12-15T23:00:09.357171+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":211,"log_dups_size":0,"ondisk_log_size":211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8991943,"num_objects":25,"num_object_clones":15,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":166,"num_read_kb":110505,"num_write":127,"num_write_kb":36801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":357,"reported_epoch":222,"state":"active+clean","last_fresh":"2023-12-15T23:00:10.543596+0000","last_change":"2023-12-15T22:56:33.655797+0000","last_active":"2023-12-15T23:00:10.543596+0000","last_peered":"2023-12-15T23:00:10.543596+0000","last_clean":"2023-12-15T23:00:10.543596+0000","last_became_active":"2023-12-15T22:56:33.652885+0000","last_became_peered":"2023-12-15T22:56:33.652885+0000","last_unstale":"2023-12-15T23:00:10.543596+0000","last_undegraded":"2023-12-15T23:00:10.543596+0000","last_fullsized":"2023-12-15T23:00:10.543596+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"221'268","reported_seq":610,"reported_epoch":222,"state":"active+clean","last_fresh":"2023-12-15T23:00:09.492774+0000","last_change":"2023-12-15T23:00:06.592426+0000","last_active":"2023-12-15T23:00:09.492774+0000","last_peered":"2023-12-15T23:00:09.492774+0000","last_clean":"2023-12-15T23:00:09.492774+0000","last_became_active":"2023-12-15T22:56:39.543509+0000","last_became_peered":"2023-12-15T22:56:39.543509+0000","last_unstale":"2023-12-15T23:00:09.492774+0000","last_undegraded":"2023-12-15T23:00:09.492774+0000","last_fullsized":"2023-12-15T23:00:09.492774+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8857487,"num_objects":27,"num_object_clones":18,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":150,"num_read_kb":111257,"num_write":112,"num_write_kb":34221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":327,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-12-15T23:00:09.359836+0000","last_change":"2023-12-15T22:56:33.600485+0000","last_active":"2023-12-15T23:00:09.359836+0000","last_peered":"2023-12-15T23:00:09.359836+0000","last_clean":"2023-12-15T23:00:09.359836+0000","last_became_active":"2023-12-15T22:56:33.597320+0000","last_became_peered":"2023-12-15T22:56:33.597320+0000","last_unstale":"2023-12-15T23:00:09.359836+0000","last_undegraded":"2023-12-15T23:00:09.359836+0000","last_fullsized":"2023-12-15T23:00:09.359836+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"200'120","reported_seq":510,"reported_epoch":222,"state":"active+clean+snaptrim","last_fresh":"2023-12-15T23:00:10.571964+0000","last_change":"2023-12-15T23:00:10.571964+0000","last_active":"2023-12-15T23:00:10.571964+0000","last_peered":"2023-12-15T23:00:10.571964+0000","last_clean":"2023-12-15T23:00:10.571964+0000","last_became_active":"2023-12-15T22:56:39.560980+0000","last_became_peered":"2023-12-15T22:56:39.560980+0000","last_unstale":"2023-12-15T23:00:10.571964+0000","last_undegraded":"2023-12-15T23:00:10.571964+0000","last_fullsized":"2023-12-15T23:00:10.571964+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6134654,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":62,"num_read_kb":52203,"num_write":49,"num_write_kb":14551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":357,"reported_epoch":222,"state":"active+clean","last_fresh":"2023-12-15T23:00:10.551712+0000","last_change":"2023-12-15T22:56:33.648541+0000","last_active":"2023-12-15T23:00:10.551712+0000","last_peered":"2023-12-15T23:00:10.551712+0000","last_clean":"2023-12-15T23:00:10.551712+0000","last_became_active":"2023-12-15T22:56:33.643681+0000","last_became_peered":"2023-12-15T22:56:33.643681+0000","last_unstale":"2023-12-15T23:00:10.551712+0000","last_undegraded":"2023-12-15T23:00:10.551712+0000","last_fullsized":"2023-12-15T23:00:10.551712+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"218'215","reported_seq":572,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-12-15T23:00:09.358184+0000","last_change":"2023-12-15T23:00:06.383985+0000","last_active":"2023-12-15T23:00:09.358184+0000","last_peered":"2023-12-15T23:00:09.358184+0000","last_clean":"2023-12-15T23:00:09.358184+0000","last_became_active":"2023-12-15T22:56:39.506510+0000","last_became_peered":"2023-12-15T22:56:39.506510+0000","last_unstale":"2023-12-15T23:00:09.358184+0000","last_undegraded":"2023-12-15T23:00:09.358184+0000","last_fullsized":"2023-12-15T23:00:09.358184+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":215,"log_dups_size":0,"ondisk_log_size":215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6269143,"num_objects":24,"num_object_clones":16,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":134,"num_read_kb":91952,"num_write":93,"num_write_kb":26998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"216'99","reported_seq":463,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-12-15T23:00:09.358918+0000","last_change":"2023-12-15T23:00:06.377813+0000","last_active":"2023-12-15T23:00:09.358918+0000","last_peered":"2023-12-15T23:00:09.358918+0000","last_clean":"2023-12-15T23:00:09.358918+0000","last_became_active":"2023-12-15T22:56:39.511159+0000","last_became_peered":"2023-12-15T22:56:39.511159+0000","last_unstale":"2023-12-15T23:00:09.358918+0000","last_undegraded":"2023-12-15T23:00:09.358918+0000","last_fullsized":"2023-12-15T23:00:09.358918+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":21,"num_read_kb":7340,"num_write":37,"num_write_kb":11102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":327,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-12-15T23:00:09.367561+0000","last_change":"2023-12-15T22:56:33.633890+0000","last_active":"2023-12-15T23:00:09.367561+0000","last_peered":"2023-12-15T23:00:09.367561+0000","last_clean":"2023-12-15T23:00:09.367561+0000","last_became_active":"2023-12-15T22:56:33.631134+0000","last_became_peered":"2023-12-15T22:56:33.631134+0000","last_unstale":"2023-12-15T23:00:09.367561+0000","last_undegraded":"2023-12-15T23:00:09.367561+0000","last_fullsized":"2023-12-15T23:00:09.367561+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"221'141","reported_seq":517,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-12-15T23:00:09.479333+0000","last_change":"2023-12-15T23:00:06.357425+0000","last_active":"2023-12-15T23:00:09.479333+0000","last_peered":"2023-12-15T23:00:09.479333+0000","last_clean":"2023-12-15T23:00:09.479333+0000","last_became_active":"2023-12-15T22:56:39.532730+0000","last_became_peered":"2023-12-15T22:56:39.532730+0000","last_unstale":"2023-12-15T23:00:09.479333+0000","last_undegraded":"2023-12-15T23:00:09.479333+0000","last_fullsized":"2023-12-15T23:00:09.479333+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5899389,"num_objects":13,"num_object_clones":9,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":76,"num_read_kb":55788,"num_write":66,"num_write_kb":20205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":341,"reported_epoch":222,"state":"active+clean","last_fresh":"2023-12-15T23:00:10.571287+0000","last_change":"2023-12-15T22:56:33.655736+0000","last_active":"2023-12-15T23:00:10.571287+0000","last_peered":"2023-12-15T23:00:10.571287+0000","last_clean":"2023-12-15T23:00:10.571287+0000","last_became_active":"2023-12-15T22:56:33.652917+0000","last_became_peered":"2023-12-15T22:56:33.652917+0000","last_unstale":"2023-12-15T23:00:10.571287+0000","last_undegraded":"2023-12-15T23:00:10.571287+0000","last_fullsized":"2023-12-15T23:00:10.571287+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"214'190","reported_seq":571,"reported_epoch":222,"state":"active+clean+snaptrim","last_fresh":"2023-12-15T23:00:10.575482+0000","last_change":"2023-12-15T23:00:10.575482+0000","last_active":"2023-12-15T23:00:10.575482+0000","last_peered":"2023-12-15T23:00:10.575482+0000","last_clean":"2023-12-15T23:00:10.575482+0000","last_became_active":"2023-12-15T22:56:39.557252+0000","last_became_peered":"2023-12-15T22:56:39.557252+0000","last_unstale":"2023-12-15T23:00:10.575482+0000","last_undegraded":"2023-12-15T23:00:10.575482+0000","last_fullsized":"2023-12-15T23:00:10.575482+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2067327,"num_objects":17,"num_object_clones":12,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":111,"num_read_kb":79710,"num_write":85,"num_write_kb":23945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":327,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-12-15T23:00:09.363287+0000","last_change":"2023-12-15T22:56:33.632348+0000","last_active":"2023-12-15T23:00:09.363287+0000","last_peered":"2023-12-15T23:00:09.363287+0000","last_clean":"2023-12-15T23:00:09.363287+0000","last_became_active":"2023-12-15T22:56:33.629022+0000","last_became_peered":"2023-12-15T22:56:33.629022+0000","last_unstale":"2023-12-15T23:00:09.363287+0000","last_undegraded":"2023-12-15T23:00:09.363287+0000","last_fullsized":"2023-12-15T23:00:09.363287+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"187'72","reported_seq":462,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-12-15T23:00:09.368376+0000","last_change":"2023-12-15T23:00:06.357600+0000","last_active":"2023-12-15T23:00:09.368376+0000","last_peered":"2023-12-15T23:00:09.368376+0000","last_clean":"2023-12-15T23:00:09.368376+0000","last_became_active":"2023-12-15T22:56:39.535145+0000","last_became_peered":"2023-12-15T22:56:39.535145+0000","last_unstale":"2023-12-15T23:00:09.368376+0000","last_undegraded":"2023-12-15T23:00:09.368376+0000","last_fullsized":"2023-12-15T23:00:09.368376+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3764801,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":31,"num_read_kb":29392,"num_write":27,"num_write_kb":8581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":327,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-12-15T23:00:09.354860+0000","last_change":"2023-12-15T22:56:33.627005+0000","last_active":"2023-12-15T23:00:09.354860+0000","last_peered":"2023-12-15T23:00:09.354860+0000","last_clean":"2023-12-15T23:00:09.354860+0000","last_became_active":"2023-12-15T22:56:33.622181+0000","last_became_peered":"2023-12-15T22:56:33.622181+0000","last_unstale":"2023-12-15T23:00:09.354860+0000","last_undegraded":"2023-12-15T23:00:09.354860+0000","last_fullsized":"2023-12-15T23:00:09.354860+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":97768068,"num_objects":298,"num_object_clones":214,"num_object_copies":894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":52,"num_read":1702,"num_read_kb":1260564,"num_write":1379,"num_write_kb":401616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3028,"ondisk_log_size":3028,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":91508,"kb_used_data":56364,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94280332,"statfs":{"total":96636764160,"available":96543059968,"internally_reserved":0,"allocated":57716736,"data_stored":57610181,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":67960,"kb_used_data":36976,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94303880,"statfs":{"total":96636764160,"available":96567173120,"internally_reserved":0,"allocated":37863424,"data_stored":37808504,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":77252,"kb_used_data":46268,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94294588,"statfs":{"total":96636764160,"available":96557658112,"internally_reserved":0,"allocated":47378432,"data_stored":46745408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":89288,"kb_used_data":54144,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94282552,"statfs":{"total":96636764160,"available":96545333248,"internally_reserved":0,"allocated":55443456,"data_stored":55400896,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-15T23:00:14.695 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T23:00:14.762 INFO:tasks.rados.rados.0.smithi071.stdout:935: finishing write tid 1 to smithi07183189-48 2023-12-15T23:00:14.762 INFO:tasks.rados.rados.0.smithi071.stdout:936: finishing write tid 1 to smithi07183189-8 2023-12-15T23:00:14.762 INFO:tasks.rados.rados.0.smithi071.stdout:936: finishing write tid 2 to smithi07183189-8 2023-12-15T23:00:14.762 INFO:tasks.rados.rados.0.smithi071.stdout:935: finishing write tid 2 to smithi07183189-48 2023-12-15T23:00:14.762 INFO:tasks.rados.rados.0.smithi071.stdout:936: finishing write tid 3 to smithi07183189-8 2023-12-15T23:00:14.762 INFO:tasks.rados.rados.0.smithi071.stdout:935: finishing write tid 3 to smithi07183189-48 2023-12-15T23:00:14.762 INFO:tasks.rados.rados.0.smithi071.stdout:936: finishing write tid 4 to smithi07183189-8 2023-12-15T23:00:14.762 INFO:tasks.rados.rados.0.smithi071.stdout:936: finishing write tid 5 to smithi07183189-8 2023-12-15T23:00:14.762 INFO:tasks.rados.rados.0.smithi071.stdout:936: oid 8 updating version 0 to 279 2023-12-15T23:00:14.762 INFO:tasks.rados.rados.0.smithi071.stdout:936: oid 8 version 279 is already newer than 276 2023-12-15T23:00:14.763 INFO:tasks.rados.rados.0.smithi071.stdout:936: oid 8 version 279 is already newer than 278 2023-12-15T23:00:14.763 INFO:tasks.rados.rados.0.smithi071.stdout:936: oid 8 version 279 is already newer than 278 2023-12-15T23:00:14.763 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 8 v 279 (ObjNum 458 snap 105 seq_num 458) dirty exists 2023-12-15T23:00:14.763 INFO:tasks.rados.rados.0.smithi071.stdout:936: left oid 8 (ObjNum 458 snap 105 seq_num 458) 2023-12-15T23:00:14.763 INFO:tasks.rados.rados.0.smithi071.stdout:935: finishing write tid 4 to smithi07183189-48 2023-12-15T23:00:14.763 INFO:tasks.rados.rados.0.smithi071.stdout:935: finishing write tid 5 to smithi07183189-48 2023-12-15T23:00:14.763 INFO:tasks.rados.rados.0.smithi071.stdout:935: oid 48 updating version 0 to 216 2023-12-15T23:00:14.763 INFO:tasks.rados.rados.0.smithi071.stdout:935: oid 48 updating version 216 to 219 2023-12-15T23:00:14.763 INFO:tasks.rados.rados.0.smithi071.stdout:935: oid 48 version 219 is already newer than 217 2023-12-15T23:00:14.763 INFO:tasks.rados.rados.0.smithi071.stdout:935: oid 48 version 219 is already newer than 218 2023-12-15T23:00:14.763 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 219 (ObjNum 457 snap 105 seq_num 457) dirty exists 2023-12-15T23:00:14.763 INFO:tasks.rados.rados.0.smithi071.stdout:935: left oid 48 (ObjNum 457 snap 105 seq_num 457) 2023-12-15T23:00:14.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:14.836+0000 7f759742d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:14.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:14.842+0000 7f759742d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:14.849 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T23:00:14.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:14.952+0000 7fddaa99e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:14.956 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:14.956+0000 7fddaa99e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:15.026 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T23:00:15.026 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-12-15T23:00:15.038 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":332,"stamp":"2023-12-15T23:00:14.554580+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98358436,"num_objects":300,"num_object_clones":214,"num_object_copies":900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":52,"num_read":1805,"num_read_kb":1260655,"num_write":1516,"num_write_kb":403510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3125,"ondisk_log_size":3125,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":326008,"kb_used_data":193752,"kb_used_omap":0,"kb_used_meta":132224,"kb_avail":377161352,"statfs":{"total":386547056640,"available":386213224448,"internally_reserved":0,"allocated":198402048,"data_stored":197564989,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":135397376},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2756348,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":17,"num_read_kb":19314,"num_write":20,"num_write_kb":6348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000589"},"pg_stats":[{"pgid":"3.f","version":"214'96","reported_seq":524,"reported_epoch":222,"state":"active+clean","last_fresh":"2023-12-15T23:00:10.540610+0000","last_change":"2023-12-15T23:00:10.558697+0000","last_active":"2023-12-15T23:00:10.540610+0000","last_peered":"2023-12-15T23:00:10.540610+0000","last_clean":"2023-12-15T23:00:10.540610+0000","last_became_active":"2023-12-15T22:56:39.503797+0000","last_became_peered":"2023-12-15T22:56:39.503797+0000","last_unstale":"2023-12-15T23:00:10.540610+0000","last_undegraded":"2023-12-15T23:00:10.540610+0000","last_fullsized":"2023-12-15T23:00:10.540610+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6084233,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":78,"num_read_kb":60669,"num_write":40,"num_write_kb":11941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":423,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-12-15T23:00:09.365470+0000","last_change":"2023-12-15T23:00:06.357073+0000","last_active":"2023-12-15T23:00:09.365470+0000","last_peered":"2023-12-15T23:00:09.365470+0000","last_clean":"2023-12-15T23:00:09.365470+0000","last_became_active":"2023-12-15T22:56:39.552856+0000","last_became_peered":"2023-12-15T22:56:39.552856+0000","last_unstale":"2023-12-15T23:00:09.365470+0000","last_undegraded":"2023-12-15T23:00:09.365470+0000","last_fullsized":"2023-12-15T23:00:09.365470+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"218'311","reported_seq":684,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-12-15T23:00:09.361441+0000","last_change":"2023-12-15T23:00:06.509587+0000","last_active":"2023-12-15T23:00:09.361441+0000","last_peered":"2023-12-15T23:00:09.361441+0000","last_clean":"2023-12-15T23:00:09.361441+0000","last_became_active":"2023-12-15T22:56:39.495490+0000","last_became_peered":"2023-12-15T22:56:39.495490+0000","last_unstale":"2023-12-15T23:00:09.361441+0000","last_undegraded":"2023-12-15T23:00:09.361441+0000","last_fullsized":"2023-12-15T23:00:09.361441+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14941687,"num_objects":33,"num_object_clones":22,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":231,"num_read_kb":202767,"num_write":169,"num_write_kb":48561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"208'291","reported_seq":653,"reported_epoch":222,"state":"active+clean","last_fresh":"2023-12-15T23:00:10.539998+0000","last_change":"2023-12-15T23:00:10.560071+0000","last_active":"2023-12-15T23:00:10.539998+0000","last_peered":"2023-12-15T23:00:10.539998+0000","last_clean":"2023-12-15T23:00:10.539998+0000","last_became_active":"2023-12-15T22:56:39.579234+0000","last_became_peered":"2023-12-15T22:56:39.579234+0000","last_unstale":"2023-12-15T23:00:10.539998+0000","last_undegraded":"2023-12-15T23:00:10.539998+0000","last_fullsized":"2023-12-15T23:00:10.539998+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8891068,"num_objects":28,"num_object_clones":24,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":155,"num_read_kb":112879,"num_write":147,"num_write_kb":42462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"215'155","reported_seq":529,"reported_epoch":222,"state":"active+clean+snaptrim","last_fresh":"2023-12-15T23:00:10.573319+0000","last_change":"2023-12-15T23:00:10.573319+0000","last_active":"2023-12-15T23:00:10.573319+0000","last_peered":"2023-12-15T23:00:10.573319+0000","last_clean":"2023-12-15T23:00:10.573319+0000","last_became_active":"2023-12-15T22:56:39.572176+0000","last_became_peered":"2023-12-15T22:56:39.572176+0000","last_unstale":"2023-12-15T23:00:10.573319+0000","last_undegraded":"2023-12-15T23:00:10.573319+0000","last_fullsized":"2023-12-15T23:00:10.573319+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7008618,"num_objects":15,"num_object_clones":10,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":66,"num_read_kb":38159,"num_write":65,"num_write_kb":19531,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"218'145","reported_seq":526,"reported_epoch":222,"state":"active+clean","last_fresh":"2023-12-15T23:00:10.555322+0000","last_change":"2023-12-15T23:00:10.559235+0000","last_active":"2023-12-15T23:00:10.555322+0000","last_peered":"2023-12-15T23:00:10.555322+0000","last_clean":"2023-12-15T23:00:10.555322+0000","last_became_active":"2023-12-15T22:56:39.516361+0000","last_became_peered":"2023-12-15T22:56:39.516361+0000","last_unstale":"2023-12-15T23:00:10.555322+0000","last_undegraded":"2023-12-15T23:00:10.555322+0000","last_fullsized":"2023-12-15T23:00:10.555322+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3327852,"num_objects":13,"num_object_clones":9,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":52,"num_read_kb":38718,"num_write":56,"num_write_kb":16218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"220'577","reported_seq":781,"reported_epoch":222,"state":"active+clean","last_fresh":"2023-12-15T23:00:10.552582+0000","last_change":"2023-12-15T23:00:10.558974+0000","last_active":"2023-12-15T23:00:10.552582+0000","last_peered":"2023-12-15T23:00:10.552582+0000","last_clean":"2023-12-15T23:00:10.552582+0000","last_became_active":"2023-12-15T22:56:39.519748+0000","last_became_peered":"2023-12-15T22:56:39.519748+0000","last_unstale":"2023-12-15T23:00:10.552582+0000","last_undegraded":"2023-12-15T23:00:10.552582+0000","last_fullsized":"2023-12-15T23:00:10.552582+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":577,"log_dups_size":0,"ondisk_log_size":577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7193495,"num_objects":53,"num_object_clones":43,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":246,"num_read_kb":159819,"num_write":232,"num_write_kb":66545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"215'137","reported_seq":545,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-12-15T23:00:09.360192+0000","last_change":"2023-12-15T23:00:06.357099+0000","last_active":"2023-12-15T23:00:09.360192+0000","last_peered":"2023-12-15T23:00:09.360192+0000","last_clean":"2023-12-15T23:00:09.360192+0000","last_became_active":"2023-12-15T22:56:39.543179+0000","last_became_peered":"2023-12-15T22:56:39.543179+0000","last_unstale":"2023-12-15T23:00:09.360192+0000","last_undegraded":"2023-12-15T23:00:09.360192+0000","last_fullsized":"2023-12-15T23:00:09.360192+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8336371,"num_objects":13,"num_object_clones":9,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":123,"num_read_kb":109406,"num_write":74,"num_write_kb":19954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":468,"reported_epoch":222,"state":"active+clean","last_fresh":"2023-12-15T23:00:09.418014+0000","last_change":"2023-12-15T22:56:32.607017+0000","last_active":"2023-12-15T23:00:09.418014+0000","last_peered":"2023-12-15T23:00:09.418014+0000","last_clean":"2023-12-15T23:00:09.418014+0000","last_became_active":"2023-12-15T22:56:32.602384+0000","last_became_peered":"2023-12-15T22:56:32.602384+0000","last_unstale":"2023-12-15T23:00:09.418014+0000","last_undegraded":"2023-12-15T23:00:09.418014+0000","last_fullsized":"2023-12-15T23:00:09.418014+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_clean_scrub_stamp":"2023-12-15T22:56:31.525572+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":327,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-12-15T23:00:09.362971+0000","last_change":"2023-12-15T22:56:33.627481+0000","last_active":"2023-12-15T23:00:09.362971+0000","last_peered":"2023-12-15T23:00:09.362971+0000","last_clean":"2023-12-15T23:00:09.362971+0000","last_became_active":"2023-12-15T22:56:33.624689+0000","last_became_peered":"2023-12-15T22:56:33.624689+0000","last_unstale":"2023-12-15T23:00:09.362971+0000","last_undegraded":"2023-12-15T23:00:09.362971+0000","last_fullsized":"2023-12-15T23:00:09.362971+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"220'211","reported_seq":617,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-12-15T23:00:09.357171+0000","last_change":"2023-12-15T23:00:06.386625+0000","last_active":"2023-12-15T23:00:09.357171+0000","last_peered":"2023-12-15T23:00:09.357171+0000","last_clean":"2023-12-15T23:00:09.357171+0000","last_became_active":"2023-12-15T22:56:39.529359+0000","last_became_peered":"2023-12-15T22:56:39.529359+0000","last_unstale":"2023-12-15T23:00:09.357171+0000","last_undegraded":"2023-12-15T23:00:09.357171+0000","last_fullsized":"2023-12-15T23:00:09.357171+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":211,"log_dups_size":0,"ondisk_log_size":211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8991943,"num_objects":25,"num_object_clones":15,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":166,"num_read_kb":110505,"num_write":127,"num_write_kb":36801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":357,"reported_epoch":222,"state":"active+clean","last_fresh":"2023-12-15T23:00:10.543596+0000","last_change":"2023-12-15T22:56:33.655797+0000","last_active":"2023-12-15T23:00:10.543596+0000","last_peered":"2023-12-15T23:00:10.543596+0000","last_clean":"2023-12-15T23:00:10.543596+0000","last_became_active":"2023-12-15T22:56:33.652885+0000","last_became_peered":"2023-12-15T22:56:33.652885+0000","last_unstale":"2023-12-15T23:00:10.543596+0000","last_undegraded":"2023-12-15T23:00:10.543596+0000","last_fullsized":"2023-12-15T23:00:10.543596+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"221'268","reported_seq":610,"reported_epoch":222,"state":"active+clean","last_fresh":"2023-12-15T23:00:09.492774+0000","last_change":"2023-12-15T23:00:06.592426+0000","last_active":"2023-12-15T23:00:09.492774+0000","last_peered":"2023-12-15T23:00:09.492774+0000","last_clean":"2023-12-15T23:00:09.492774+0000","last_became_active":"2023-12-15T22:56:39.543509+0000","last_became_peered":"2023-12-15T22:56:39.543509+0000","last_unstale":"2023-12-15T23:00:09.492774+0000","last_undegraded":"2023-12-15T23:00:09.492774+0000","last_fullsized":"2023-12-15T23:00:09.492774+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8857487,"num_objects":27,"num_object_clones":18,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":150,"num_read_kb":111257,"num_write":112,"num_write_kb":34221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":327,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-12-15T23:00:09.359836+0000","last_change":"2023-12-15T22:56:33.600485+0000","last_active":"2023-12-15T23:00:09.359836+0000","last_peered":"2023-12-15T23:00:09.359836+0000","last_clean":"2023-12-15T23:00:09.359836+0000","last_became_active":"2023-12-15T22:56:33.597320+0000","last_became_peered":"2023-12-15T22:56:33.597320+0000","last_unstale":"2023-12-15T23:00:09.359836+0000","last_undegraded":"2023-12-15T23:00:09.359836+0000","last_fullsized":"2023-12-15T23:00:09.359836+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"200'120","reported_seq":510,"reported_epoch":222,"state":"active+clean+snaptrim","last_fresh":"2023-12-15T23:00:10.571964+0000","last_change":"2023-12-15T23:00:10.571964+0000","last_active":"2023-12-15T23:00:10.571964+0000","last_peered":"2023-12-15T23:00:10.571964+0000","last_clean":"2023-12-15T23:00:10.571964+0000","last_became_active":"2023-12-15T22:56:39.560980+0000","last_became_peered":"2023-12-15T22:56:39.560980+0000","last_unstale":"2023-12-15T23:00:10.571964+0000","last_undegraded":"2023-12-15T23:00:10.571964+0000","last_fullsized":"2023-12-15T23:00:10.571964+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6134654,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":62,"num_read_kb":52203,"num_write":49,"num_write_kb":14551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":357,"reported_epoch":222,"state":"active+clean","last_fresh":"2023-12-15T23:00:10.551712+0000","last_change":"2023-12-15T22:56:33.648541+0000","last_active":"2023-12-15T23:00:10.551712+0000","last_peered":"2023-12-15T23:00:10.551712+0000","last_clean":"2023-12-15T23:00:10.551712+0000","last_became_active":"2023-12-15T22:56:33.643681+0000","last_became_peered":"2023-12-15T22:56:33.643681+0000","last_unstale":"2023-12-15T23:00:10.551712+0000","last_undegraded":"2023-12-15T23:00:10.551712+0000","last_fullsized":"2023-12-15T23:00:10.551712+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"218'215","reported_seq":572,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-12-15T23:00:09.358184+0000","last_change":"2023-12-15T23:00:06.383985+0000","last_active":"2023-12-15T23:00:09.358184+0000","last_peered":"2023-12-15T23:00:09.358184+0000","last_clean":"2023-12-15T23:00:09.358184+0000","last_became_active":"2023-12-15T22:56:39.506510+0000","last_became_peered":"2023-12-15T22:56:39.506510+0000","last_unstale":"2023-12-15T23:00:09.358184+0000","last_undegraded":"2023-12-15T23:00:09.358184+0000","last_fullsized":"2023-12-15T23:00:09.358184+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":215,"log_dups_size":0,"ondisk_log_size":215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6269143,"num_objects":24,"num_object_clones":16,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":134,"num_read_kb":91952,"num_write":93,"num_write_kb":26998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"216'99","reported_seq":463,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-12-15T23:00:09.358918+0000","last_change":"2023-12-15T23:00:06.377813+0000","last_active":"2023-12-15T23:00:09.358918+0000","last_peered":"2023-12-15T23:00:09.358918+0000","last_clean":"2023-12-15T23:00:09.358918+0000","last_became_active":"2023-12-15T22:56:39.511159+0000","last_became_peered":"2023-12-15T22:56:39.511159+0000","last_unstale":"2023-12-15T23:00:09.358918+0000","last_undegraded":"2023-12-15T23:00:09.358918+0000","last_fullsized":"2023-12-15T23:00:09.358918+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":21,"num_read_kb":7340,"num_write":37,"num_write_kb":11102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":327,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-12-15T23:00:09.367561+0000","last_change":"2023-12-15T22:56:33.633890+0000","last_active":"2023-12-15T23:00:09.367561+0000","last_peered":"2023-12-15T23:00:09.367561+0000","last_clean":"2023-12-15T23:00:09.367561+0000","last_became_active":"2023-12-15T22:56:33.631134+0000","last_became_peered":"2023-12-15T22:56:33.631134+0000","last_unstale":"2023-12-15T23:00:09.367561+0000","last_undegraded":"2023-12-15T23:00:09.367561+0000","last_fullsized":"2023-12-15T23:00:09.367561+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"221'141","reported_seq":517,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-12-15T23:00:09.479333+0000","last_change":"2023-12-15T23:00:06.357425+0000","last_active":"2023-12-15T23:00:09.479333+0000","last_peered":"2023-12-15T23:00:09.479333+0000","last_clean":"2023-12-15T23:00:09.479333+0000","last_became_active":"2023-12-15T22:56:39.532730+0000","last_became_peered":"2023-12-15T22:56:39.532730+0000","last_unstale":"2023-12-15T23:00:09.479333+0000","last_undegraded":"2023-12-15T23:00:09.479333+0000","last_fullsized":"2023-12-15T23:00:09.479333+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5899389,"num_objects":13,"num_object_clones":9,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":76,"num_read_kb":55788,"num_write":66,"num_write_kb":20205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":341,"reported_epoch":222,"state":"active+clean","last_fresh":"2023-12-15T23:00:10.571287+0000","last_change":"2023-12-15T22:56:33.655736+0000","last_active":"2023-12-15T23:00:10.571287+0000","last_peered":"2023-12-15T23:00:10.571287+0000","last_clean":"2023-12-15T23:00:10.571287+0000","last_became_active":"2023-12-15T22:56:33.652917+0000","last_became_peered":"2023-12-15T22:56:33.652917+0000","last_unstale":"2023-12-15T23:00:10.571287+0000","last_undegraded":"2023-12-15T23:00:10.571287+0000","last_fullsized":"2023-12-15T23:00:10.571287+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"214'190","reported_seq":571,"reported_epoch":222,"state":"active+clean+snaptrim","last_fresh":"2023-12-15T23:00:10.575482+0000","last_change":"2023-12-15T23:00:10.575482+0000","last_active":"2023-12-15T23:00:10.575482+0000","last_peered":"2023-12-15T23:00:10.575482+0000","last_clean":"2023-12-15T23:00:10.575482+0000","last_became_active":"2023-12-15T22:56:39.557252+0000","last_became_peered":"2023-12-15T22:56:39.557252+0000","last_unstale":"2023-12-15T23:00:10.575482+0000","last_undegraded":"2023-12-15T23:00:10.575482+0000","last_fullsized":"2023-12-15T23:00:10.575482+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2067327,"num_objects":17,"num_object_clones":12,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":111,"num_read_kb":79710,"num_write":85,"num_write_kb":23945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":327,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-12-15T23:00:09.363287+0000","last_change":"2023-12-15T22:56:33.632348+0000","last_active":"2023-12-15T23:00:09.363287+0000","last_peered":"2023-12-15T23:00:09.363287+0000","last_clean":"2023-12-15T23:00:09.363287+0000","last_became_active":"2023-12-15T22:56:33.629022+0000","last_became_peered":"2023-12-15T22:56:33.629022+0000","last_unstale":"2023-12-15T23:00:09.363287+0000","last_undegraded":"2023-12-15T23:00:09.363287+0000","last_fullsized":"2023-12-15T23:00:09.363287+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"187'72","reported_seq":462,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-12-15T23:00:09.368376+0000","last_change":"2023-12-15T23:00:06.357600+0000","last_active":"2023-12-15T23:00:09.368376+0000","last_peered":"2023-12-15T23:00:09.368376+0000","last_clean":"2023-12-15T23:00:09.368376+0000","last_became_active":"2023-12-15T22:56:39.535145+0000","last_became_peered":"2023-12-15T22:56:39.535145+0000","last_unstale":"2023-12-15T23:00:09.368376+0000","last_undegraded":"2023-12-15T23:00:09.368376+0000","last_fullsized":"2023-12-15T23:00:09.368376+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3764801,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":31,"num_read_kb":29392,"num_write":27,"num_write_kb":8581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":327,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-12-15T23:00:09.354860+0000","last_change":"2023-12-15T22:56:33.627005+0000","last_active":"2023-12-15T23:00:09.354860+0000","last_peered":"2023-12-15T23:00:09.354860+0000","last_clean":"2023-12-15T23:00:09.354860+0000","last_became_active":"2023-12-15T22:56:33.622181+0000","last_became_peered":"2023-12-15T22:56:33.622181+0000","last_unstale":"2023-12-15T23:00:09.354860+0000","last_undegraded":"2023-12-15T23:00:09.354860+0000","last_fullsized":"2023-12-15T23:00:09.354860+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":97768068,"num_objects":298,"num_object_clones":214,"num_object_copies":894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":52,"num_read":1702,"num_read_kb":1260564,"num_write":1379,"num_write_kb":401616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3028,"ondisk_log_size":3028,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":91508,"kb_used_data":56364,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94280332,"statfs":{"total":96636764160,"available":96543059968,"internally_reserved":0,"allocated":57716736,"data_stored":57610181,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":67960,"kb_used_data":36976,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94303880,"statfs":{"total":96636764160,"available":96567173120,"internally_reserved":0,"allocated":37863424,"data_stored":37808504,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":77252,"kb_used_data":46268,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94294588,"statfs":{"total":96636764160,"available":96557658112,"internally_reserved":0,"allocated":47378432,"data_stored":46745408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":89288,"kb_used_data":54144,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94282552,"statfs":{"total":96636764160,"available":96545333248,"internally_reserved":0,"allocated":55443456,"data_stored":55400896,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-15T23:00:15.039 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2023-12-15T23:00:15.039 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T23:00:15.039 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-12-15T23:00:15.040 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-12-15T23:00:15.184 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:15.179+0000 7fb7d8413640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:15.187 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:15.187+0000 7fb7d8413640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:15.199 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T23:00:15.199 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T23:00:15.199 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T23:00:15.199 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T23:00:15.199 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T23:00:15.199 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T23:00:15.199 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T23:00:15.200 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T23:00:15.200 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T23:00:15.200 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T23:00:15.200 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T23:00:15.200 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T23:00:15.202 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T23:00:15.202 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:00:15.202 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-12-15T23:00:15.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:15.304+0000 7f08d8f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:15.312 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:15.311+0000 7f08d8f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:15.324 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T23:00:15.324 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T23:00:15.324 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T23:00:15.324 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T23:00:15.324 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T23:00:15.324 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T23:00:15.324 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T23:00:15.324 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T23:00:15.324 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T23:00:15.324 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T23:00:15.324 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T23:00:15.324 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T23:00:15.327 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T23:00:15.327 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:00:15.327 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-12-15T23:00:15.370 INFO:tasks.rados.rados.0.smithi071.stdout:932: done (5 left) 2023-12-15T23:00:15.370 INFO:tasks.rados.rados.0.smithi071.stdout:933: done (4 left) 2023-12-15T23:00:15.370 INFO:tasks.rados.rados.0.smithi071.stdout:934: done (3 left) 2023-12-15T23:00:15.371 INFO:tasks.rados.rados.0.smithi071.stdout:935: done (2 left) 2023-12-15T23:00:15.371 INFO:tasks.rados.rados.0.smithi071.stdout:936: done (1 left) 2023-12-15T23:00:15.371 INFO:tasks.rados.rados.0.smithi071.stdout:937: done (0 left) 2023-12-15T23:00:15.371 INFO:tasks.rados.rados.0.smithi071.stdout:938: snap_create 2023-12-15T23:00:15.435 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:15.430+0000 7f065cf1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:15.438 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:15.438+0000 7f065cf1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:15.451 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T23:00:15.451 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T23:00:15.451 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T23:00:15.451 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T23:00:15.451 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T23:00:15.451 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T23:00:15.451 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T23:00:15.451 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T23:00:15.451 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T23:00:15.451 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T23:00:15.451 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T23:00:15.451 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T23:00:15.454 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T23:00:15.454 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:00:15.454 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-12-15T23:00:15.569 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:15.565+0000 7ffb41f04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:15.573 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:15.573+0000 7ffb41f04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:15.585 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T23:00:15.585 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T23:00:15.585 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T23:00:15.585 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T23:00:15.586 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T23:00:15.586 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T23:00:15.586 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T23:00:15.586 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T23:00:15.586 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T23:00:15.586 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T23:00:15.586 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T23:00:15.586 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T23:00:15.588 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T23:00:15.588 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:00:16.372 INFO:tasks.rados.rados.0.smithi071.stdout:938: done (0 left) 2023-12-15T23:00:16.372 INFO:tasks.rados.rados.0.smithi071.stdout:939: write (excl) oid 44 current snap is 107 2023-12-15T23:00:16.372 INFO:tasks.rados.rados.0.smithi071.stdout:939: seq_num 459 ranges {514413=720471,1876529=773152,3438683=275730} 2023-12-15T23:00:16.378 INFO:tasks.rados.rados.0.smithi071.stdout:939: writing smithi07183189-44 from 514413 to 1234884 tid 1 2023-12-15T23:00:16.386 INFO:tasks.rados.rados.0.smithi071.stdout:939: writing smithi07183189-44 from 1876529 to 2649681 tid 2 2023-12-15T23:00:16.392 INFO:tasks.rados.rados.0.smithi071.stdout:939: writing smithi07183189-44 from 3438683 to 3714413 tid 3 2023-12-15T23:00:16.392 INFO:tasks.rados.rados.0.smithi071.stdout:940: read oid 11 snap 105 2023-12-15T23:00:16.392 INFO:tasks.rados.rados.0.smithi071.stdout:940: expect (ObjNum 390 snap 95 seq_num 390) 2023-12-15T23:00:16.393 INFO:tasks.rados.rados.0.smithi071.stdout:941: read oid 48 snap -1 2023-12-15T23:00:16.393 INFO:tasks.rados.rados.0.smithi071.stdout:941: expect (ObjNum 457 snap 105 seq_num 457) 2023-12-15T23:00:16.393 INFO:tasks.rados.rados.0.smithi071.stdout:942: snap_remove snap 103 2023-12-15T23:00:17.029 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T23:00:17.135 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:17.134+0000 7f7acad1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:17.138 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:17.138+0000 7f7acad1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:17.379 INFO:tasks.rados.rados.0.smithi071.stdout:942: done (3 left) 2023-12-15T23:00:17.379 INFO:tasks.rados.rados.0.smithi071.stdout:943: snap_remove snap 106 2023-12-15T23:00:18.381 INFO:tasks.rados.rados.0.smithi071.stdout:943: done (3 left) 2023-12-15T23:00:18.381 INFO:tasks.rados.rados.0.smithi071.stdout:944: write oid 19 current snap is 107 2023-12-15T23:00:18.381 INFO:tasks.rados.rados.0.smithi071.stdout:944: seq_num 460 ranges {531220=795720,1935011=516810,3066106=420420} 2023-12-15T23:00:18.388 INFO:tasks.rados.rados.0.smithi071.stdout:944: writing smithi07183189-19 from 531220 to 1326940 tid 1 2023-12-15T23:00:18.394 INFO:tasks.rados.rados.0.smithi071.stdout:944: writing smithi07183189-19 from 1935011 to 2451821 tid 2 2023-12-15T23:00:18.402 INFO:tasks.rados.rados.0.smithi071.stdout:944: writing smithi07183189-19 from 3066106 to 3486526 tid 3 2023-12-15T23:00:18.402 INFO:tasks.rados.rados.0.smithi071.stdout:945: snap_remove snap 104 2023-12-15T23:00:19.209 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T23:00:19.318 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:19.317+0000 7fd322d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:19.321 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:19.321+0000 7fd322d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:19.390 INFO:tasks.rados.rados.0.smithi071.stdout:945: done (4 left) 2023-12-15T23:00:19.390 INFO:tasks.rados.rados.0.smithi071.stdout:946: snap_create 2023-12-15T23:00:19.390 INFO:tasks.rados.rados.0.smithi071.stdout:940: expect (ObjNum 390 snap 95 seq_num 390) 2023-12-15T23:00:19.502 INFO:tasks.rados.rados.0.smithi071.stdout:941: expect (ObjNum 457 snap 105 seq_num 457) 2023-12-15T23:00:19.645 INFO:tasks.rados.rados.0.smithi071.stdout:939: finishing write tid 1 to smithi07183189-44 2023-12-15T23:00:19.645 INFO:tasks.rados.rados.0.smithi071.stdout:939: finishing write tid 2 to smithi07183189-44 2023-12-15T23:00:19.645 INFO:tasks.rados.rados.0.smithi071.stdout:939: finishing write tid 3 to smithi07183189-44 2023-12-15T23:00:19.645 INFO:tasks.rados.rados.0.smithi071.stdout:939: finishing write tid 4 to smithi07183189-44 2023-12-15T23:00:19.645 INFO:tasks.rados.rados.0.smithi071.stdout:939: finishing write tid 5 to smithi07183189-44 2023-12-15T23:00:19.645 INFO:tasks.rados.rados.0.smithi071.stdout:939: oid 44 updating version 0 to 145 2023-12-15T23:00:19.645 INFO:tasks.rados.rados.0.smithi071.stdout:939: oid 44 version 145 is already newer than 145 2023-12-15T23:00:19.646 INFO:tasks.rados.rados.0.smithi071.stdout:939: oid 44 updating version 145 to 146 2023-12-15T23:00:19.646 INFO:tasks.rados.rados.0.smithi071.stdout:939: oid 44 version 146 is already newer than 143 2023-12-15T23:00:19.646 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 146 (ObjNum 459 snap 107 seq_num 459) dirty exists 2023-12-15T23:00:19.646 INFO:tasks.rados.rados.0.smithi071.stdout:939: left oid 44 (ObjNum 459 snap 107 seq_num 459) 2023-12-15T23:00:19.646 INFO:tasks.rados.rados.0.smithi071.stdout:944: finishing write tid 1 to smithi07183189-19 2023-12-15T23:00:19.646 INFO:tasks.rados.rados.0.smithi071.stdout:944: finishing write tid 2 to smithi07183189-19 2023-12-15T23:00:19.646 INFO:tasks.rados.rados.0.smithi071.stdout:944: finishing write tid 3 to smithi07183189-19 2023-12-15T23:00:19.646 INFO:tasks.rados.rados.0.smithi071.stdout:944: finishing write tid 4 to smithi07183189-19 2023-12-15T23:00:19.646 INFO:tasks.rados.rados.0.smithi071.stdout:944: finishing write tid 5 to smithi07183189-19 2023-12-15T23:00:19.646 INFO:tasks.rados.rados.0.smithi071.stdout:944: oid 19 updating version 0 to 304 2023-12-15T23:00:19.646 INFO:tasks.rados.rados.0.smithi071.stdout:944: oid 19 version 304 is already newer than 301 2023-12-15T23:00:19.646 INFO:tasks.rados.rados.0.smithi071.stdout:944: oid 19 version 304 is already newer than 303 2023-12-15T23:00:19.646 INFO:tasks.rados.rados.0.smithi071.stdout:944: oid 19 version 304 is already newer than 302 2023-12-15T23:00:19.646 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 19 v 304 (ObjNum 460 snap 107 seq_num 460) dirty exists 2023-12-15T23:00:19.646 INFO:tasks.rados.rados.0.smithi071.stdout:944: left oid 19 (ObjNum 460 snap 107 seq_num 460) 2023-12-15T23:00:20.390 INFO:tasks.rados.rados.0.smithi071.stdout:939: done (4 left) 2023-12-15T23:00:20.391 INFO:tasks.rados.rados.0.smithi071.stdout:940: done (3 left) 2023-12-15T23:00:20.391 INFO:tasks.rados.rados.0.smithi071.stdout:941: done (2 left) 2023-12-15T23:00:20.391 INFO:tasks.rados.rados.0.smithi071.stdout:944: done (1 left) 2023-12-15T23:00:20.391 INFO:tasks.rados.rados.0.smithi071.stdout:946: done (0 left) 2023-12-15T23:00:20.391 INFO:tasks.rados.rados.0.smithi071.stdout:947: snap_remove snap 105 2023-12-15T23:00:20.589 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T23:00:20.589 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T23:00:20.589 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T23:00:20.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:20.694+0000 7f371cb5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:20.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:20.700+0000 7f371cb5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:20.872 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T23:00:20.873 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-12-15T23:00:20.884 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":341,"stamp":"2023-12-15T23:00:20.555586+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":100913232,"num_objects":310,"num_object_clones":222,"num_object_copies":930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":55,"num_read":1864,"num_read_kb":1303144,"num_write":1571,"num_write_kb":421224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3205,"ondisk_log_size":3205,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":323956,"kb_used_data":191120,"kb_used_omap":0,"kb_used_meta":132224,"kb_avail":377163404,"statfs":{"total":386547056640,"available":386215325696,"internally_reserved":0,"allocated":195706880,"data_stored":199959944,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":135397376},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1243718,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":15253,"num_write":5,"num_write_kb":1730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000686"},"pg_stats":[{"pgid":"3.f","version":"214'96","reported_seq":532,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-12-15T23:00:15.547088+0000","last_change":"2023-12-15T23:00:12.310532+0000","last_active":"2023-12-15T23:00:15.547088+0000","last_peered":"2023-12-15T23:00:15.547088+0000","last_clean":"2023-12-15T23:00:15.547088+0000","last_became_active":"2023-12-15T22:56:39.503797+0000","last_became_peered":"2023-12-15T22:56:39.503797+0000","last_unstale":"2023-12-15T23:00:15.547088+0000","last_undegraded":"2023-12-15T23:00:15.547088+0000","last_fullsized":"2023-12-15T23:00:15.547088+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6084233,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":78,"num_read_kb":60669,"num_write":40,"num_write_kb":11941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":439,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-12-15T23:00:17.396832+0000","last_change":"2023-12-15T23:00:17.401627+0000","last_active":"2023-12-15T23:00:17.396832+0000","last_peered":"2023-12-15T23:00:17.396832+0000","last_clean":"2023-12-15T23:00:17.396832+0000","last_became_active":"2023-12-15T22:56:39.552856+0000","last_became_peered":"2023-12-15T22:56:39.552856+0000","last_unstale":"2023-12-15T23:00:17.396832+0000","last_undegraded":"2023-12-15T23:00:17.396832+0000","last_fullsized":"2023-12-15T23:00:17.396832+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"218'311","reported_seq":707,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-12-15T23:00:17.639195+0000","last_change":"2023-12-15T23:00:17.647783+0000","last_active":"2023-12-15T23:00:17.639195+0000","last_peered":"2023-12-15T23:00:17.639195+0000","last_clean":"2023-12-15T23:00:17.639195+0000","last_became_active":"2023-12-15T22:56:39.495490+0000","last_became_peered":"2023-12-15T22:56:39.495490+0000","last_unstale":"2023-12-15T23:00:17.639195+0000","last_undegraded":"2023-12-15T23:00:17.639195+0000","last_fullsized":"2023-12-15T23:00:17.639195+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14941687,"num_objects":33,"num_object_clones":22,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":237,"num_read_kb":206908,"num_write":169,"num_write_kb":48561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"225'297","reported_seq":667,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-12-15T23:00:15.543207+0000","last_change":"2023-12-15T23:00:12.312082+0000","last_active":"2023-12-15T23:00:15.543207+0000","last_peered":"2023-12-15T23:00:15.543207+0000","last_clean":"2023-12-15T23:00:15.543207+0000","last_became_active":"2023-12-15T22:56:39.579234+0000","last_became_peered":"2023-12-15T22:56:39.579234+0000","last_unstale":"2023-12-15T23:00:15.543207+0000","last_undegraded":"2023-12-15T23:00:15.543207+0000","last_fullsized":"2023-12-15T23:00:15.543207+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":297,"log_dups_size":0,"ondisk_log_size":297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6033845,"num_objects":28,"num_object_clones":25,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":158,"num_read_kb":112880,"num_write":152,"num_write_kb":44283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"225'161","reported_seq":546,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-12-15T23:00:15.541990+0000","last_change":"2023-12-15T23:00:12.352265+0000","last_active":"2023-12-15T23:00:15.541990+0000","last_peered":"2023-12-15T23:00:15.541990+0000","last_clean":"2023-12-15T23:00:15.541990+0000","last_became_active":"2023-12-15T22:56:39.572176+0000","last_became_peered":"2023-12-15T22:56:39.572176+0000","last_unstale":"2023-12-15T23:00:15.541990+0000","last_undegraded":"2023-12-15T23:00:15.541990+0000","last_fullsized":"2023-12-15T23:00:15.541990+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6554862,"num_objects":17,"num_object_clones":11,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":68,"num_read_kb":38160,"num_write":70,"num_write_kb":21200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"218'145","reported_seq":534,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-12-15T23:00:15.537888+0000","last_change":"2023-12-15T23:00:12.320144+0000","last_active":"2023-12-15T23:00:15.537888+0000","last_peered":"2023-12-15T23:00:15.537888+0000","last_clean":"2023-12-15T23:00:15.537888+0000","last_became_active":"2023-12-15T22:56:39.516361+0000","last_became_peered":"2023-12-15T22:56:39.516361+0000","last_unstale":"2023-12-15T23:00:15.537888+0000","last_undegraded":"2023-12-15T23:00:15.537888+0000","last_fullsized":"2023-12-15T23:00:15.537888+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3327852,"num_objects":13,"num_object_clones":9,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":52,"num_read_kb":38718,"num_write":56,"num_write_kb":16218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"225'588","reported_seq":799,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-12-15T23:00:15.546361+0000","last_change":"2023-12-15T23:00:13.401116+0000","last_active":"2023-12-15T23:00:15.546361+0000","last_peered":"2023-12-15T23:00:15.546361+0000","last_clean":"2023-12-15T23:00:15.546361+0000","last_became_active":"2023-12-15T22:56:39.519748+0000","last_became_peered":"2023-12-15T22:56:39.519748+0000","last_unstale":"2023-12-15T23:00:15.546361+0000","last_undegraded":"2023-12-15T23:00:15.546361+0000","last_fullsized":"2023-12-15T23:00:15.546361+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":588,"log_dups_size":0,"ondisk_log_size":588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11059171,"num_objects":55,"num_object_clones":44,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":249,"num_read_kb":159821,"num_write":242,"num_write_kb":69439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"229'141","reported_seq":566,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-12-15T23:00:17.639034+0000","last_change":"2023-12-15T23:00:17.663161+0000","last_active":"2023-12-15T23:00:17.639034+0000","last_peered":"2023-12-15T23:00:17.639034+0000","last_clean":"2023-12-15T23:00:17.639034+0000","last_became_active":"2023-12-15T22:56:39.543179+0000","last_became_peered":"2023-12-15T22:56:39.543179+0000","last_unstale":"2023-12-15T23:00:17.639034+0000","last_undegraded":"2023-12-15T23:00:17.639034+0000","last_fullsized":"2023-12-15T23:00:17.639034+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5428793,"num_objects":14,"num_object_clones":10,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":124,"num_read_kb":109406,"num_write":74,"num_write_kb":19954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":478,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-12-15T23:00:15.541760+0000","last_change":"2023-12-15T22:56:32.607017+0000","last_active":"2023-12-15T23:00:15.541760+0000","last_peered":"2023-12-15T23:00:15.541760+0000","last_clean":"2023-12-15T23:00:15.541760+0000","last_became_active":"2023-12-15T22:56:32.602384+0000","last_became_peered":"2023-12-15T22:56:32.602384+0000","last_unstale":"2023-12-15T23:00:15.541760+0000","last_undegraded":"2023-12-15T23:00:15.541760+0000","last_fullsized":"2023-12-15T23:00:15.541760+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_clean_scrub_stamp":"2023-12-15T22:56:31.525572+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":343,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-12-15T23:00:17.635088+0000","last_change":"2023-12-15T22:56:33.627481+0000","last_active":"2023-12-15T23:00:17.635088+0000","last_peered":"2023-12-15T23:00:17.635088+0000","last_clean":"2023-12-15T23:00:17.635088+0000","last_became_active":"2023-12-15T22:56:33.624689+0000","last_became_peered":"2023-12-15T22:56:33.624689+0000","last_unstale":"2023-12-15T23:00:17.635088+0000","last_undegraded":"2023-12-15T23:00:17.635088+0000","last_fullsized":"2023-12-15T23:00:17.635088+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"226'219","reported_seq":651,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-12-15T23:00:17.636049+0000","last_change":"2023-12-15T23:00:17.640805+0000","last_active":"2023-12-15T23:00:17.636049+0000","last_peered":"2023-12-15T23:00:17.636049+0000","last_clean":"2023-12-15T23:00:17.636049+0000","last_became_active":"2023-12-15T22:56:39.529359+0000","last_became_peered":"2023-12-15T22:56:39.529359+0000","last_unstale":"2023-12-15T23:00:17.636049+0000","last_undegraded":"2023-12-15T23:00:17.636049+0000","last_fullsized":"2023-12-15T23:00:17.636049+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":219,"log_dups_size":0,"ondisk_log_size":219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12672742,"num_objects":26,"num_object_clones":15,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":186,"num_read_kb":134232,"num_write":132,"num_write_kb":38683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":363,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-12-15T23:00:15.541322+0000","last_change":"2023-12-15T22:56:33.655797+0000","last_active":"2023-12-15T23:00:15.541322+0000","last_peered":"2023-12-15T23:00:15.541322+0000","last_clean":"2023-12-15T23:00:15.541322+0000","last_became_active":"2023-12-15T22:56:33.652885+0000","last_became_peered":"2023-12-15T22:56:33.652885+0000","last_unstale":"2023-12-15T23:00:15.541322+0000","last_undegraded":"2023-12-15T23:00:15.541322+0000","last_fullsized":"2023-12-15T23:00:15.541322+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"226'279","reported_seq":632,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-12-15T23:00:15.539388+0000","last_change":"2023-12-15T23:00:13.484652+0000","last_active":"2023-12-15T23:00:15.539388+0000","last_peered":"2023-12-15T23:00:15.539388+0000","last_clean":"2023-12-15T23:00:15.539388+0000","last_became_active":"2023-12-15T22:56:39.543509+0000","last_became_peered":"2023-12-15T22:56:39.543509+0000","last_unstale":"2023-12-15T23:00:15.539388+0000","last_undegraded":"2023-12-15T23:00:15.539388+0000","last_fullsized":"2023-12-15T23:00:15.539388+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5832161,"num_objects":29,"num_object_clones":20,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":159,"num_read_kb":116905,"num_write":117,"num_write_kb":35911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":343,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-12-15T23:00:17.636234+0000","last_change":"2023-12-15T22:56:33.600485+0000","last_active":"2023-12-15T23:00:17.636234+0000","last_peered":"2023-12-15T23:00:17.636234+0000","last_clean":"2023-12-15T23:00:17.636234+0000","last_became_active":"2023-12-15T22:56:33.597320+0000","last_became_peered":"2023-12-15T22:56:33.597320+0000","last_unstale":"2023-12-15T23:00:17.636234+0000","last_undegraded":"2023-12-15T23:00:17.636234+0000","last_fullsized":"2023-12-15T23:00:17.636234+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"224'124","reported_seq":526,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-12-15T23:00:15.541314+0000","last_change":"2023-12-15T23:00:12.354807+0000","last_active":"2023-12-15T23:00:15.541314+0000","last_peered":"2023-12-15T23:00:15.541314+0000","last_clean":"2023-12-15T23:00:15.541314+0000","last_became_active":"2023-12-15T22:56:39.560980+0000","last_became_peered":"2023-12-15T22:56:39.560980+0000","last_unstale":"2023-12-15T23:00:15.541314+0000","last_undegraded":"2023-12-15T23:00:15.541314+0000","last_fullsized":"2023-12-15T23:00:15.541314+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344758,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":64,"num_read_kb":52204,"num_write":54,"num_write_kb":16040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":363,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-12-15T23:00:15.543700+0000","last_change":"2023-12-15T22:56:33.648541+0000","last_active":"2023-12-15T23:00:15.543700+0000","last_peered":"2023-12-15T23:00:15.543700+0000","last_clean":"2023-12-15T23:00:15.543700+0000","last_became_active":"2023-12-15T22:56:33.643681+0000","last_became_peered":"2023-12-15T22:56:33.643681+0000","last_unstale":"2023-12-15T23:00:15.543700+0000","last_undegraded":"2023-12-15T23:00:15.543700+0000","last_fullsized":"2023-12-15T23:00:15.543700+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"229'228","reported_seq":596,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-12-15T23:00:17.392703+0000","last_change":"2023-12-15T23:00:17.426190+0000","last_active":"2023-12-15T23:00:17.392703+0000","last_peered":"2023-12-15T23:00:17.392703+0000","last_clean":"2023-12-15T23:00:17.392703+0000","last_became_active":"2023-12-15T22:56:39.506510+0000","last_became_peered":"2023-12-15T22:56:39.506510+0000","last_unstale":"2023-12-15T23:00:17.392703+0000","last_undegraded":"2023-12-15T23:00:17.392703+0000","last_fullsized":"2023-12-15T23:00:17.392703+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":228,"log_dups_size":0,"ondisk_log_size":228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6823774,"num_objects":25,"num_object_clones":17,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":142,"num_read_kb":100917,"num_write":98,"num_write_kb":28699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"224'103","reported_seq":488,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-12-15T23:00:17.635606+0000","last_change":"2023-12-15T23:00:17.643258+0000","last_active":"2023-12-15T23:00:17.635606+0000","last_peered":"2023-12-15T23:00:17.635606+0000","last_clean":"2023-12-15T23:00:17.635606+0000","last_became_active":"2023-12-15T22:56:39.511159+0000","last_became_peered":"2023-12-15T22:56:39.511159+0000","last_unstale":"2023-12-15T23:00:17.635606+0000","last_undegraded":"2023-12-15T23:00:17.635606+0000","last_fullsized":"2023-12-15T23:00:17.635606+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3479115,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":22,"num_read_kb":7341,"num_write":42,"num_write_kb":12640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":339,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-12-15T23:00:17.394249+0000","last_change":"2023-12-15T22:56:33.633890+0000","last_active":"2023-12-15T23:00:17.394249+0000","last_peered":"2023-12-15T23:00:17.394249+0000","last_clean":"2023-12-15T23:00:17.394249+0000","last_became_active":"2023-12-15T22:56:33.631134+0000","last_became_peered":"2023-12-15T22:56:33.631134+0000","last_unstale":"2023-12-15T23:00:17.394249+0000","last_undegraded":"2023-12-15T23:00:17.394249+0000","last_fullsized":"2023-12-15T23:00:17.394249+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"229'148","reported_seq":538,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-12-15T23:00:17.388738+0000","last_change":"2023-12-15T23:00:17.424210+0000","last_active":"2023-12-15T23:00:17.388738+0000","last_peered":"2023-12-15T23:00:17.388738+0000","last_clean":"2023-12-15T23:00:17.388738+0000","last_became_active":"2023-12-15T22:56:39.532730+0000","last_became_peered":"2023-12-15T22:56:39.532730+0000","last_unstale":"2023-12-15T23:00:17.388738+0000","last_undegraded":"2023-12-15T23:00:17.388738+0000","last_fullsized":"2023-12-15T23:00:17.388738+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143107,"num_objects":14,"num_object_clones":10,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":78,"num_read_kb":55789,"num_write":71,"num_write_kb":21935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":349,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-12-15T23:00:15.542069+0000","last_change":"2023-12-15T22:56:33.655736+0000","last_active":"2023-12-15T23:00:15.542069+0000","last_peered":"2023-12-15T23:00:15.542069+0000","last_clean":"2023-12-15T23:00:15.542069+0000","last_became_active":"2023-12-15T22:56:33.652917+0000","last_became_peered":"2023-12-15T22:56:33.652917+0000","last_unstale":"2023-12-15T23:00:15.542069+0000","last_undegraded":"2023-12-15T23:00:15.542069+0000","last_fullsized":"2023-12-15T23:00:15.542069+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"225'195","reported_seq":588,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-12-15T23:00:15.535243+0000","last_change":"2023-12-15T23:00:12.354166+0000","last_active":"2023-12-15T23:00:15.535243+0000","last_peered":"2023-12-15T23:00:15.535243+0000","last_clean":"2023-12-15T23:00:15.535243+0000","last_became_active":"2023-12-15T22:56:39.557252+0000","last_became_peered":"2023-12-15T22:56:39.557252+0000","last_unstale":"2023-12-15T23:00:15.535243+0000","last_undegraded":"2023-12-15T23:00:15.535243+0000","last_fullsized":"2023-12-15T23:00:15.535243+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3596764,"num_objects":17,"num_object_clones":12,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":113,"num_read_kb":79711,"num_write":90,"num_write_kb":25245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":339,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-12-15T23:00:17.388587+0000","last_change":"2023-12-15T22:56:33.632348+0000","last_active":"2023-12-15T23:00:17.388587+0000","last_peered":"2023-12-15T23:00:17.388587+0000","last_clean":"2023-12-15T23:00:17.388587+0000","last_became_active":"2023-12-15T22:56:33.629022+0000","last_became_peered":"2023-12-15T22:56:33.629022+0000","last_unstale":"2023-12-15T23:00:17.388587+0000","last_undegraded":"2023-12-15T23:00:17.388587+0000","last_fullsized":"2023-12-15T23:00:17.388587+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"224'73","reported_seq":479,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-12-15T23:00:17.393378+0000","last_change":"2023-12-15T23:00:17.402108+0000","last_active":"2023-12-15T23:00:17.393378+0000","last_peered":"2023-12-15T23:00:17.393378+0000","last_clean":"2023-12-15T23:00:17.393378+0000","last_became_active":"2023-12-15T22:56:39.535145+0000","last_became_peered":"2023-12-15T22:56:39.535145+0000","last_unstale":"2023-12-15T23:00:17.393378+0000","last_undegraded":"2023-12-15T23:00:17.393378+0000","last_fullsized":"2023-12-15T23:00:17.393378+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":31,"num_read_kb":29392,"num_write":27,"num_write_kb":8581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":343,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-12-15T23:00:17.637880+0000","last_change":"2023-12-15T22:56:33.627005+0000","last_active":"2023-12-15T23:00:17.637880+0000","last_peered":"2023-12-15T23:00:17.637880+0000","last_clean":"2023-12-15T23:00:17.637880+0000","last_became_active":"2023-12-15T22:56:33.622181+0000","last_became_peered":"2023-12-15T22:56:33.622181+0000","last_unstale":"2023-12-15T23:00:17.637880+0000","last_undegraded":"2023-12-15T23:00:17.637880+0000","last_fullsized":"2023-12-15T23:00:17.637880+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":100322864,"num_objects":308,"num_object_clones":222,"num_object_copies":924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":55,"num_read":1761,"num_read_kb":1303053,"num_write":1434,"num_write_kb":419330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3108,"ondisk_log_size":3108,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705889,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":92540,"kb_used_data":56816,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94279300,"statfs":{"total":96636764160,"available":96542003200,"internally_reserved":0,"allocated":58179584,"data_stored":59014332,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":66660,"kb_used_data":35676,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94305180,"statfs":{"total":96636764160,"available":96568504320,"internally_reserved":0,"allocated":36532224,"data_stored":37884449,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":73920,"kb_used_data":42936,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94297920,"statfs":{"total":96636764160,"available":96561070080,"internally_reserved":0,"allocated":43966464,"data_stored":44000018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705889,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":90836,"kb_used_data":55692,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94281004,"statfs":{"total":96636764160,"available":96543748096,"internally_reserved":0,"allocated":57028608,"data_stored":59061145,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-15T23:00:21.394 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T23:00:21.398 INFO:tasks.rados.rados.0.smithi071.stdout:947: done (0 left) 2023-12-15T23:00:21.398 INFO:tasks.rados.rados.0.smithi071.stdout:948: delete oid 19 current snap is 108 2023-12-15T23:00:21.473 INFO:tasks.rados.rados.0.smithi071.stdout:948: done (0 left) 2023-12-15T23:00:21.473 INFO:tasks.rados.rados.0.smithi071.stdout:949: write oid 48 current snap is 108 2023-12-15T23:00:21.473 INFO:tasks.rados.rados.0.smithi071.stdout:949: seq_num 462 ranges {564834=546218,1651975=520479,2720952=559317} 2023-12-15T23:00:21.479 INFO:tasks.rados.rados.0.smithi071.stdout:949: writing smithi07183189-48 from 564834 to 1111052 tid 1 2023-12-15T23:00:21.485 INFO:tasks.rados.rados.0.smithi071.stdout:949: writing smithi07183189-48 from 1651975 to 2172454 tid 2 2023-12-15T23:00:21.492 INFO:tasks.rados.rados.0.smithi071.stdout:949: writing smithi07183189-48 from 2720952 to 3280269 tid 3 2023-12-15T23:00:21.492 INFO:tasks.rados.rados.0.smithi071.stdout:950: delete oid 50 current snap is 108 2023-12-15T23:00:21.510 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:21.508+0000 7f7accc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:21.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:21.513+0000 7f7accc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:21.561 INFO:tasks.rados.rados.0.smithi071.stdout:949: finishing write tid 1 to smithi07183189-48 2023-12-15T23:00:21.563 INFO:tasks.rados.rados.0.smithi071.stdout:950: done (1 left) 2023-12-15T23:00:21.564 INFO:tasks.rados.rados.0.smithi071.stdout:951: snap_remove snap 107 2023-12-15T23:00:22.275 INFO:tasks.rados.rados.0.smithi071.stdout:951: done (1 left) 2023-12-15T23:00:22.275 INFO:tasks.rados.rados.0.smithi071.stdout:952: write oid 47 current snap is 108 2023-12-15T23:00:22.275 INFO:tasks.rados.rados.0.smithi071.stdout:952: seq_num 464 ranges {598448=413069,1768939=524148,2775798=581861,3781488=16960} 2023-12-15T23:00:22.280 INFO:tasks.rados.rados.0.smithi071.stdout:952: writing smithi07183189-47 from 598448 to 1011517 tid 1 2023-12-15T23:00:22.287 INFO:tasks.rados.rados.0.smithi071.stdout:952: writing smithi07183189-47 from 1768939 to 2293087 tid 2 2023-12-15T23:00:22.293 INFO:tasks.rados.rados.0.smithi071.stdout:952: writing smithi07183189-47 from 2775798 to 3357659 tid 3 2023-12-15T23:00:22.296 INFO:tasks.rados.rados.0.smithi071.stdout:952: writing smithi07183189-47 from 3781488 to 3798448 tid 4 2023-12-15T23:00:22.296 INFO:tasks.rados.rados.0.smithi071.stdout:953: write (excl) oid 38 current snap is 108 2023-12-15T23:00:22.296 INFO:tasks.rados.rados.0.smithi071.stdout:953: seq_num 465 ranges {615255=488318,1827421=784159,3203221=612034} 2023-12-15T23:00:22.302 INFO:tasks.rados.rados.0.smithi071.stdout:953: writing smithi07183189-38 from 615255 to 1103573 tid 1 2023-12-15T23:00:22.311 INFO:tasks.rados.rados.0.smithi071.stdout:953: writing smithi07183189-38 from 1827421 to 2611580 tid 2 2023-12-15T23:00:22.318 INFO:tasks.rados.rados.0.smithi071.stdout:953: writing smithi07183189-38 from 3203221 to 3815255 tid 3 2023-12-15T23:00:22.318 INFO:tasks.rados.rados.0.smithi071.stdout:954: read oid 31 snap -1 2023-12-15T23:00:22.318 INFO:tasks.rados.rados.0.smithi071.stdout:954: expect deleted 2023-12-15T23:00:22.319 INFO:tasks.rados.rados.0.smithi071.stdout:955: read oid 4 snap -1 2023-12-15T23:00:22.319 INFO:tasks.rados.rados.0.smithi071.stdout:955: expect (ObjNum 422 snap 99 seq_num 422) 2023-12-15T23:00:22.319 INFO:tasks.rados.rados.0.smithi071.stdout:956: read oid 1 snap -1 2023-12-15T23:00:22.319 INFO:tasks.rados.rados.0.smithi071.stdout:956: expect (ObjNum 407 snap 98 seq_num 407) 2023-12-15T23:00:22.319 INFO:tasks.rados.rados.0.smithi071.stdout:957: delete oid 11 current snap is 108 2023-12-15T23:00:22.319 INFO:tasks.rados.rados.0.smithi071.stdout:949: finishing write tid 2 to smithi07183189-48 2023-12-15T23:00:22.319 INFO:tasks.rados.rados.0.smithi071.stdout:949: finishing write tid 3 to smithi07183189-48 2023-12-15T23:00:22.319 INFO:tasks.rados.rados.0.smithi071.stdout:949: finishing write tid 4 to smithi07183189-48 2023-12-15T23:00:22.319 INFO:tasks.rados.rados.0.smithi071.stdout:949: finishing write tid 5 to smithi07183189-48 2023-12-15T23:00:22.319 INFO:tasks.rados.rados.0.smithi071.stdout:949: oid 48 updating version 0 to 222 2023-12-15T23:00:22.319 INFO:tasks.rados.rados.0.smithi071.stdout:949: oid 48 updating version 222 to 224 2023-12-15T23:00:22.319 INFO:tasks.rados.rados.0.smithi071.stdout:949: oid 48 version 224 is already newer than 221 2023-12-15T23:00:22.319 INFO:tasks.rados.rados.0.smithi071.stdout:949: oid 48 version 224 is already newer than 224 2023-12-15T23:00:22.319 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 224 (ObjNum 462 snap 108 seq_num 462) dirty exists 2023-12-15T23:00:22.320 INFO:tasks.rados.rados.0.smithi071.stdout:949: left oid 48 (ObjNum 462 snap 108 seq_num 462) 2023-12-15T23:00:22.402 INFO:tasks.rados.rados.0.smithi071.stdout:956: expect (ObjNum 407 snap 98 seq_num 407) 2023-12-15T23:00:22.566 INFO:tasks.rados.rados.0.smithi071.stdout:955: expect (ObjNum 422 snap 99 seq_num 422) 2023-12-15T23:00:22.763 INFO:tasks.rados.rados.0.smithi071.stdout:953: finishing write tid 1 to smithi07183189-38 2023-12-15T23:00:22.764 INFO:tasks.rados.rados.0.smithi071.stdout:952: finishing write tid 1 to smithi07183189-47 2023-12-15T23:00:22.764 INFO:tasks.rados.rados.0.smithi071.stdout:952: finishing write tid 2 to smithi07183189-47 2023-12-15T23:00:22.764 INFO:tasks.rados.rados.0.smithi071.stdout:953: finishing write tid 2 to smithi07183189-38 2023-12-15T23:00:22.764 INFO:tasks.rados.rados.0.smithi071.stdout:953: finishing write tid 3 to smithi07183189-38 2023-12-15T23:00:22.764 INFO:tasks.rados.rados.0.smithi071.stdout:952: finishing write tid 3 to smithi07183189-47 2023-12-15T23:00:22.764 INFO:tasks.rados.rados.0.smithi071.stdout:953: finishing write tid 4 to smithi07183189-38 2023-12-15T23:00:22.764 INFO:tasks.rados.rados.0.smithi071.stdout:953: finishing write tid 5 to smithi07183189-38 2023-12-15T23:00:22.764 INFO:tasks.rados.rados.0.smithi071.stdout:953: oid 38 updating version 0 to 144 2023-12-15T23:00:22.764 INFO:tasks.rados.rados.0.smithi071.stdout:953: oid 38 version 144 is already newer than 143 2023-12-15T23:00:22.764 INFO:tasks.rados.rados.0.smithi071.stdout:953: oid 38 updating version 144 to 145 2023-12-15T23:00:22.764 INFO:tasks.rados.rados.0.smithi071.stdout:953: oid 38 version 145 is already newer than 142 2023-12-15T23:00:22.764 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 38 v 145 (ObjNum 465 snap 108 seq_num 465) dirty exists 2023-12-15T23:00:22.764 INFO:tasks.rados.rados.0.smithi071.stdout:953: left oid 38 (ObjNum 465 snap 108 seq_num 465) 2023-12-15T23:00:22.765 INFO:tasks.rados.rados.0.smithi071.stdout:952: finishing write tid 4 to smithi07183189-47 2023-12-15T23:00:22.765 INFO:tasks.rados.rados.0.smithi071.stdout:952: finishing write tid 5 to smithi07183189-47 2023-12-15T23:00:22.765 INFO:tasks.rados.rados.0.smithi071.stdout:952: finishing write tid 6 to smithi07183189-47 2023-12-15T23:00:22.765 INFO:tasks.rados.rados.0.smithi071.stdout:952: oid 47 updating version 0 to 167 2023-12-15T23:00:22.765 INFO:tasks.rados.rados.0.smithi071.stdout:952: oid 47 version 167 is already newer than 166 2023-12-15T23:00:22.765 INFO:tasks.rados.rados.0.smithi071.stdout:952: oid 47 updating version 167 to 168 2023-12-15T23:00:22.765 INFO:tasks.rados.rados.0.smithi071.stdout:952: oid 47 version 168 is already newer than 165 2023-12-15T23:00:22.765 INFO:tasks.rados.rados.0.smithi071.stdout:952: oid 47 version 168 is already newer than 164 2023-12-15T23:00:22.765 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 168 (ObjNum 464 snap 108 seq_num 464) dirty exists 2023-12-15T23:00:22.765 INFO:tasks.rados.rados.0.smithi071.stdout:952: left oid 47 (ObjNum 464 snap 108 seq_num 464) 2023-12-15T23:00:22.765 INFO:tasks.rados.rados.0.smithi071.stdout:949: done (6 left) 2023-12-15T23:00:22.765 INFO:tasks.rados.rados.0.smithi071.stdout:952: done (5 left) 2023-12-15T23:00:22.765 INFO:tasks.rados.rados.0.smithi071.stdout:953: done (4 left) 2023-12-15T23:00:22.765 INFO:tasks.rados.rados.0.smithi071.stdout:954: done (3 left) 2023-12-15T23:00:22.766 INFO:tasks.rados.rados.0.smithi071.stdout:955: done (2 left) 2023-12-15T23:00:22.766 INFO:tasks.rados.rados.0.smithi071.stdout:956: done (1 left) 2023-12-15T23:00:22.766 INFO:tasks.rados.rados.0.smithi071.stdout:957: done (0 left) 2023-12-15T23:00:22.766 INFO:tasks.rados.rados.0.smithi071.stdout:958: snap_create 2023-12-15T23:00:23.403 INFO:tasks.rados.rados.0.smithi071.stdout:958: done (0 left) 2023-12-15T23:00:23.403 INFO:tasks.rados.rados.0.smithi071.stdout:959: snap_create 2023-12-15T23:00:23.631 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T23:00:23.736 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:23.735+0000 7f1f6fd5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:23.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:23.739+0000 7f1f6fd5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:24.405 INFO:tasks.rados.rados.0.smithi071.stdout:959: done (0 left) 2023-12-15T23:00:24.405 INFO:tasks.rados.rados.0.smithi071.stdout:960: write (excl) oid 44 current snap is 110 2023-12-15T23:00:24.405 INFO:tasks.rados.rados.0.smithi071.stdout:960: seq_num 467 ranges {648869=638816,1828032=787828,3141714=532030} 2023-12-15T23:00:24.412 INFO:tasks.rados.rados.0.smithi071.stdout:960: writing smithi07183189-44 from 648869 to 1287685 tid 1 2023-12-15T23:00:24.419 INFO:tasks.rados.rados.0.smithi071.stdout:960: writing smithi07183189-44 from 1828032 to 2615860 tid 2 2023-12-15T23:00:24.425 INFO:tasks.rados.rados.0.smithi071.stdout:960: writing smithi07183189-44 from 3141714 to 3673744 tid 3 2023-12-15T23:00:24.425 INFO:tasks.rados.rados.0.smithi071.stdout:961: delete oid 38 current snap is 110 2023-12-15T23:00:24.489 INFO:tasks.rados.rados.0.smithi071.stdout:961: done (1 left) 2023-12-15T23:00:24.489 INFO:tasks.rados.rados.0.smithi071.stdout:962: write (excl) oid 13 current snap is 110 2023-12-15T23:00:24.489 INFO:tasks.rados.rados.0.smithi071.stdout:962: seq_num 469 ranges {682483=789314,1944996=791497,3196560=554574} 2023-12-15T23:00:24.497 INFO:tasks.rados.rados.0.smithi071.stdout:962: writing smithi07183189-13 from 682483 to 1471797 tid 1 2023-12-15T23:00:24.504 INFO:tasks.rados.rados.0.smithi071.stdout:962: writing smithi07183189-13 from 1944996 to 2736493 tid 2 2023-12-15T23:00:24.510 INFO:tasks.rados.rados.0.smithi071.stdout:962: writing smithi07183189-13 from 3196560 to 3751134 tid 3 2023-12-15T23:00:24.510 INFO:tasks.rados.rados.0.smithi071.stdout:963: delete oid 17 current snap is 110 2023-12-15T23:00:24.510 INFO:tasks.rados.rados.0.smithi071.stdout:960: finishing write tid 1 to smithi07183189-44 2023-12-15T23:00:24.514 INFO:tasks.rados.rados.0.smithi071.stdout:960: finishing write tid 2 to smithi07183189-44 2023-12-15T23:00:24.541 INFO:tasks.rados.rados.0.smithi071.stdout:963: done (2 left) 2023-12-15T23:00:24.541 INFO:tasks.rados.rados.0.smithi071.stdout:964: write oid 18 current snap is 110 2023-12-15T23:00:24.541 INFO:tasks.rados.rados.0.smithi071.stdout:964: seq_num 471 ranges {716097=539812,1945607=795166,3535053=381044} 2023-12-15T23:00:24.548 INFO:tasks.rados.rados.0.smithi071.stdout:964: writing smithi07183189-18 from 716097 to 1255909 tid 1 2023-12-15T23:00:24.557 INFO:tasks.rados.rados.0.smithi071.stdout:964: writing smithi07183189-18 from 1945607 to 2740773 tid 2 2023-12-15T23:00:24.565 INFO:tasks.rados.rados.0.smithi071.stdout:964: writing smithi07183189-18 from 3535053 to 3916097 tid 3 2023-12-15T23:00:24.565 INFO:tasks.rados.rados.0.smithi071.stdout:965: read oid 42 snap -1 2023-12-15T23:00:24.565 INFO:tasks.rados.rados.0.smithi071.stdout:965: expect (ObjNum 428 snap 100 seq_num 428) 2023-12-15T23:00:24.565 INFO:tasks.rados.rados.0.smithi071.stdout:966: write (excl) oid 24 current snap is 110 2023-12-15T23:00:24.565 INFO:tasks.rados.rados.0.smithi071.stdout:960: finishing write tid 3 to smithi07183189-44 2023-12-15T23:00:24.565 INFO:tasks.rados.rados.0.smithi071.stdout:966: seq_num 472 ranges {732904=731414,2120442=538824,3278829=504743} 2023-12-15T23:00:24.574 INFO:tasks.rados.rados.0.smithi071.stdout:966: writing smithi07183189-24 from 732904 to 1464318 tid 1 2023-12-15T23:00:24.580 INFO:tasks.rados.rados.0.smithi071.stdout:966: writing smithi07183189-24 from 2120442 to 2659266 tid 2 2023-12-15T23:00:24.587 INFO:tasks.rados.rados.0.smithi071.stdout:966: writing smithi07183189-24 from 3278829 to 3783572 tid 3 2023-12-15T23:00:24.587 INFO:tasks.rados.rados.0.smithi071.stdout:967: write oid 27 current snap is 110 2023-12-15T23:00:24.587 INFO:tasks.rados.rados.0.smithi071.stdout:967: seq_num 473 ranges {749711=406663,1778924=682482,3189899=716015} 2023-12-15T23:00:24.593 INFO:tasks.rados.rados.0.smithi071.stdout:967: writing smithi07183189-27 from 749711 to 1156374 tid 1 2023-12-15T23:00:24.600 INFO:tasks.rados.rados.0.smithi071.stdout:967: writing smithi07183189-27 from 1778924 to 2461406 tid 2 2023-12-15T23:00:24.609 INFO:tasks.rados.rados.0.smithi071.stdout:967: writing smithi07183189-27 from 3189899 to 3905914 tid 3 2023-12-15T23:00:24.609 INFO:tasks.rados.rados.0.smithi071.stdout:968: snap_create 2023-12-15T23:00:24.609 INFO:tasks.rados.rados.0.smithi071.stdout:960: finishing write tid 4 to smithi07183189-44 2023-12-15T23:00:24.609 INFO:tasks.rados.rados.0.smithi071.stdout:960: finishing write tid 5 to smithi07183189-44 2023-12-15T23:00:24.609 INFO:tasks.rados.rados.0.smithi071.stdout:960: oid 44 updating version 0 to 152 2023-12-15T23:00:24.610 INFO:tasks.rados.rados.0.smithi071.stdout:960: oid 44 updating version 152 to 155 2023-12-15T23:00:24.610 INFO:tasks.rados.rados.0.smithi071.stdout:960: oid 44 version 155 is already newer than 153 2023-12-15T23:00:24.610 INFO:tasks.rados.rados.0.smithi071.stdout:960: oid 44 version 155 is already newer than 155 2023-12-15T23:00:24.610 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 44 v 155 (ObjNum 467 snap 110 seq_num 467) dirty exists 2023-12-15T23:00:24.610 INFO:tasks.rados.rados.0.smithi071.stdout:960: left oid 44 (ObjNum 467 snap 110 seq_num 467) 2023-12-15T23:00:24.610 INFO:tasks.rados.rados.0.smithi071.stdout:962: finishing write tid 1 to smithi07183189-13 2023-12-15T23:00:24.610 INFO:tasks.rados.rados.0.smithi071.stdout:962: finishing write tid 2 to smithi07183189-13 2023-12-15T23:00:24.610 INFO:tasks.rados.rados.0.smithi071.stdout:965: expect (ObjNum 428 snap 100 seq_num 428) 2023-12-15T23:00:24.784 INFO:tasks.rados.rados.0.smithi071.stdout:962: finishing write tid 3 to smithi07183189-13 2023-12-15T23:00:24.784 INFO:tasks.rados.rados.0.smithi071.stdout:966: finishing write tid 1 to smithi07183189-24 2023-12-15T23:00:24.784 INFO:tasks.rados.rados.0.smithi071.stdout:964: finishing write tid 1 to smithi07183189-18 2023-12-15T23:00:24.784 INFO:tasks.rados.rados.0.smithi071.stdout:967: finishing write tid 1 to smithi07183189-27 2023-12-15T23:00:24.784 INFO:tasks.rados.rados.0.smithi071.stdout:966: finishing write tid 2 to smithi07183189-24 2023-12-15T23:00:24.784 INFO:tasks.rados.rados.0.smithi071.stdout:962: finishing write tid 4 to smithi07183189-13 2023-12-15T23:00:24.784 INFO:tasks.rados.rados.0.smithi071.stdout:962: finishing write tid 5 to smithi07183189-13 2023-12-15T23:00:24.784 INFO:tasks.rados.rados.0.smithi071.stdout:962: oid 13 updating version 0 to 289 2023-12-15T23:00:24.784 INFO:tasks.rados.rados.0.smithi071.stdout:962: oid 13 version 289 is already newer than 286 2023-12-15T23:00:24.784 INFO:tasks.rados.rados.0.smithi071.stdout:962: oid 13 version 289 is already newer than 288 2023-12-15T23:00:24.785 INFO:tasks.rados.rados.0.smithi071.stdout:962: oid 13 version 289 is already newer than 288 2023-12-15T23:00:24.785 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 289 (ObjNum 469 snap 110 seq_num 469) dirty exists 2023-12-15T23:00:24.785 INFO:tasks.rados.rados.0.smithi071.stdout:962: left oid 13 (ObjNum 469 snap 110 seq_num 469) 2023-12-15T23:00:24.785 INFO:tasks.rados.rados.0.smithi071.stdout:966: finishing write tid 3 to smithi07183189-24 2023-12-15T23:00:24.785 INFO:tasks.rados.rados.0.smithi071.stdout:964: finishing write tid 2 to smithi07183189-18 2023-12-15T23:00:24.785 INFO:tasks.rados.rados.0.smithi071.stdout:967: finishing write tid 2 to smithi07183189-27 2023-12-15T23:00:24.785 INFO:tasks.rados.rados.0.smithi071.stdout:964: finishing write tid 3 to smithi07183189-18 2023-12-15T23:00:24.785 INFO:tasks.rados.rados.0.smithi071.stdout:967: finishing write tid 3 to smithi07183189-27 2023-12-15T23:00:24.785 INFO:tasks.rados.rados.0.smithi071.stdout:966: finishing write tid 4 to smithi07183189-24 2023-12-15T23:00:24.786 INFO:tasks.rados.rados.0.smithi071.stdout:966: finishing write tid 5 to smithi07183189-24 2023-12-15T23:00:24.786 INFO:tasks.rados.rados.0.smithi071.stdout:966: oid 24 updating version 0 to 105 2023-12-15T23:00:24.786 INFO:tasks.rados.rados.0.smithi071.stdout:966: oid 24 updating version 105 to 107 2023-12-15T23:00:24.786 INFO:tasks.rados.rados.0.smithi071.stdout:966: oid 24 version 107 is already newer than 107 2023-12-15T23:00:24.786 INFO:tasks.rados.rados.0.smithi071.stdout:966: oid 24 updating version 107 to 108 2023-12-15T23:00:24.786 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 24 v 108 (ObjNum 472 snap 110 seq_num 472) dirty exists 2023-12-15T23:00:24.786 INFO:tasks.rados.rados.0.smithi071.stdout:966: left oid 24 (ObjNum 472 snap 110 seq_num 472) 2023-12-15T23:00:24.786 INFO:tasks.rados.rados.0.smithi071.stdout:964: finishing write tid 4 to smithi07183189-18 2023-12-15T23:00:24.786 INFO:tasks.rados.rados.0.smithi071.stdout:964: finishing write tid 5 to smithi07183189-18 2023-12-15T23:00:24.786 INFO:tasks.rados.rados.0.smithi071.stdout:964: oid 18 updating version 0 to 146 2023-12-15T23:00:24.786 INFO:tasks.rados.rados.0.smithi071.stdout:964: oid 18 updating version 146 to 148 2023-12-15T23:00:24.786 INFO:tasks.rados.rados.0.smithi071.stdout:964: oid 18 updating version 148 to 149 2023-12-15T23:00:24.786 INFO:tasks.rados.rados.0.smithi071.stdout:964: oid 18 version 149 is already newer than 148 2023-12-15T23:00:24.786 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 149 (ObjNum 471 snap 110 seq_num 471) dirty exists 2023-12-15T23:00:24.786 INFO:tasks.rados.rados.0.smithi071.stdout:964: left oid 18 (ObjNum 471 snap 110 seq_num 471) 2023-12-15T23:00:24.786 INFO:tasks.rados.rados.0.smithi071.stdout:967: finishing write tid 4 to smithi07183189-27 2023-12-15T23:00:24.786 INFO:tasks.rados.rados.0.smithi071.stdout:967: finishing write tid 5 to smithi07183189-27 2023-12-15T23:00:24.787 INFO:tasks.rados.rados.0.smithi071.stdout:967: oid 27 updating version 0 to 233 2023-12-15T23:00:24.787 INFO:tasks.rados.rados.0.smithi071.stdout:967: oid 27 version 233 is already newer than 230 2023-12-15T23:00:24.787 INFO:tasks.rados.rados.0.smithi071.stdout:967: oid 27 version 233 is already newer than 231 2023-12-15T23:00:24.787 INFO:tasks.rados.rados.0.smithi071.stdout:967: oid 27 version 233 is already newer than 232 2023-12-15T23:00:24.787 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 233 (ObjNum 473 snap 110 seq_num 473) dirty exists 2023-12-15T23:00:24.787 INFO:tasks.rados.rados.0.smithi071.stdout:967: left oid 27 (ObjNum 473 snap 110 seq_num 473) 2023-12-15T23:00:25.412 INFO:tasks.rados.rados.0.smithi071.stdout:960: done (6 left) 2023-12-15T23:00:25.412 INFO:tasks.rados.rados.0.smithi071.stdout:962: done (5 left) 2023-12-15T23:00:25.412 INFO:tasks.rados.rados.0.smithi071.stdout:964: done (4 left) 2023-12-15T23:00:25.412 INFO:tasks.rados.rados.0.smithi071.stdout:965: done (3 left) 2023-12-15T23:00:25.412 INFO:tasks.rados.rados.0.smithi071.stdout:966: done (2 left) 2023-12-15T23:00:25.412 INFO:tasks.rados.rados.0.smithi071.stdout:967: done (1 left) 2023-12-15T23:00:25.412 INFO:tasks.rados.rados.0.smithi071.stdout:968: done (0 left) 2023-12-15T23:00:25.413 INFO:tasks.rados.rados.0.smithi071.stdout:969: write oid 46 current snap is 111 2023-12-15T23:00:25.413 INFO:tasks.rados.rados.0.smithi071.stdout:969: seq_num 474 ranges {766518=481912,1837406=542493,2933675=527287,3876584=89934} 2023-12-15T23:00:25.418 INFO:tasks.rados.rados.0.smithi071.stdout:969: writing smithi07183189-46 from 766518 to 1248430 tid 1 2023-12-15T23:00:25.425 INFO:tasks.rados.rados.0.smithi071.stdout:969: writing smithi07183189-46 from 1837406 to 2379899 tid 2 2023-12-15T23:00:25.431 INFO:tasks.rados.rados.0.smithi071.stdout:969: writing smithi07183189-46 from 2933675 to 3460962 tid 3 2023-12-15T23:00:25.434 INFO:tasks.rados.rados.0.smithi071.stdout:969: writing smithi07183189-46 from 3876584 to 3966518 tid 4 2023-12-15T23:00:25.434 INFO:tasks.rados.rados.0.smithi071.stdout:970: read oid 45 snap 109 2023-12-15T23:00:25.434 INFO:tasks.rados.rados.0.smithi071.stdout:970: expect deleted 2023-12-15T23:00:25.434 INFO:tasks.rados.rados.0.smithi071.stdout:971: snap_create 2023-12-15T23:00:25.451 INFO:tasks.rados.rados.0.smithi071.stdout:969: finishing write tid 1 to smithi07183189-46 2023-12-15T23:00:25.468 INFO:tasks.rados.rados.0.smithi071.stdout:969: finishing write tid 2 to smithi07183189-46 2023-12-15T23:00:25.484 INFO:tasks.rados.rados.0.smithi071.stdout:969: finishing write tid 3 to smithi07183189-46 2023-12-15T23:00:25.495 INFO:tasks.rados.rados.0.smithi071.stdout:969: finishing write tid 4 to smithi07183189-46 2023-12-15T23:00:25.497 INFO:tasks.rados.rados.0.smithi071.stdout:969: finishing write tid 5 to smithi07183189-46 2023-12-15T23:00:25.498 INFO:tasks.rados.rados.0.smithi071.stdout:969: finishing write tid 6 to smithi07183189-46 2023-12-15T23:00:25.498 INFO:tasks.rados.rados.0.smithi071.stdout:969: oid 46 updating version 0 to 157 2023-12-15T23:00:25.499 INFO:tasks.rados.rados.0.smithi071.stdout:969: oid 46 updating version 157 to 161 2023-12-15T23:00:25.499 INFO:tasks.rados.rados.0.smithi071.stdout:969: oid 46 version 161 is already newer than 161 2023-12-15T23:00:25.499 INFO:tasks.rados.rados.0.smithi071.stdout:969: oid 46 version 161 is already newer than 159 2023-12-15T23:00:25.499 INFO:tasks.rados.rados.0.smithi071.stdout:969: oid 46 version 161 is already newer than 158 2023-12-15T23:00:25.499 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 46 v 161 (ObjNum 474 snap 111 seq_num 474) dirty exists 2023-12-15T23:00:25.499 INFO:tasks.rados.rados.0.smithi071.stdout:969: left oid 46 (ObjNum 474 snap 111 seq_num 474) 2023-12-15T23:00:25.807 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T23:00:25.885 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T23:00:25.885 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T23:00:25.886 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-12-15T23:00:25.886 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-12-15T23:00:25.915 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:25.913+0000 7f1c4af9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:25.917 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:25.918+0000 7f1c4af9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:26.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:26.003+0000 7fea3271e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:26.011 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:26.011+0000 7fea3271e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:26.023 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T23:00:26.023 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T23:00:26.023 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T23:00:26.023 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T23:00:26.023 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T23:00:26.023 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T23:00:26.023 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T23:00:26.023 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T23:00:26.023 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T23:00:26.023 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T23:00:26.023 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T23:00:26.023 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T23:00:26.025 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T23:00:26.026 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:00:26.026 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-12-15T23:00:26.172 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:26.167+0000 7f1f06c0f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:26.175 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:26.175+0000 7f1f06c0f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:26.187 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T23:00:26.187 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T23:00:26.187 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T23:00:26.187 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T23:00:26.187 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T23:00:26.187 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T23:00:26.187 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T23:00:26.187 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T23:00:26.187 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T23:00:26.187 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T23:00:26.187 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T23:00:26.187 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T23:00:26.189 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T23:00:26.190 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:00:26.190 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-12-15T23:00:26.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:26.290+0000 7f6ce795e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:26.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:26.298+0000 7f6ce795e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:26.310 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T23:00:26.310 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T23:00:26.310 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T23:00:26.310 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T23:00:26.310 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T23:00:26.310 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T23:00:26.310 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T23:00:26.310 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T23:00:26.310 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T23:00:26.311 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T23:00:26.311 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T23:00:26.311 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T23:00:26.312 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T23:00:26.313 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:00:26.313 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-12-15T23:00:26.415 INFO:tasks.rados.rados.0.smithi071.stdout:969: done (2 left) 2023-12-15T23:00:26.415 INFO:tasks.rados.rados.0.smithi071.stdout:970: done (1 left) 2023-12-15T23:00:26.415 INFO:tasks.rados.rados.0.smithi071.stdout:971: done (0 left) 2023-12-15T23:00:26.415 INFO:tasks.rados.rados.0.smithi071.stdout:972: read oid 26 snap 109 2023-12-15T23:00:26.415 INFO:tasks.rados.rados.0.smithi071.stdout:972: expect (ObjNum 452 snap 104 seq_num 452) 2023-12-15T23:00:26.415 INFO:tasks.rados.rados.0.smithi071.stdout:973: snap_remove snap 111 2023-12-15T23:00:26.419 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:26.415+0000 7f970a82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:26.422 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:26.422+0000 7f970a82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:26.449 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T23:00:26.449 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T23:00:26.450 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T23:00:26.450 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T23:00:26.450 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T23:00:26.450 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T23:00:26.450 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T23:00:26.450 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T23:00:26.450 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T23:00:26.450 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T23:00:26.450 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T23:00:26.450 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T23:00:26.452 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T23:00:26.453 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:00:27.420 INFO:tasks.rados.rados.0.smithi071.stdout:973: done (1 left) 2023-12-15T23:00:27.420 INFO:tasks.rados.rados.0.smithi071.stdout:974: delete oid 28 current snap is 112 2023-12-15T23:00:27.420 INFO:tasks.rados.rados.0.smithi071.stdout:972: expect (ObjNum 452 snap 104 seq_num 452) 2023-12-15T23:00:27.583 INFO:tasks.rados.rados.0.smithi071.stdout:972: done (1 left) 2023-12-15T23:00:27.583 INFO:tasks.rados.rados.0.smithi071.stdout:974: done (0 left) 2023-12-15T23:00:27.583 INFO:tasks.rados.rados.0.smithi071.stdout:975: snap_remove snap 110 2023-12-15T23:00:27.999 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T23:00:28.104 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:28.103+0000 7fdf236ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:28.107 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:28.107+0000 7fdf236ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:28.420 INFO:tasks.rados.rados.0.smithi071.stdout:975: done (0 left) 2023-12-15T23:00:28.420 INFO:tasks.rados.rados.0.smithi071.stdout:976: delete oid 19 current snap is 112 2023-12-15T23:00:28.459 INFO:tasks.rados.rados.0.smithi071.stdout:976: done (0 left) 2023-12-15T23:00:28.459 INFO:tasks.rados.rados.0.smithi071.stdout:977: snap_remove snap 109 2023-12-15T23:00:29.428 INFO:tasks.rados.rados.0.smithi071.stdout:977: done (0 left) 2023-12-15T23:00:29.428 INFO:tasks.rados.rados.0.smithi071.stdout:978: snap_remove snap 108 2023-12-15T23:00:30.177 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T23:00:30.292 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:30.291+0000 7f54e2ade640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:30.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:30.295+0000 7f54e2ade640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:30.429 INFO:tasks.rados.rados.0.smithi071.stdout:978: done (0 left) 2023-12-15T23:00:30.429 INFO:tasks.rados.rados.0.smithi071.stdout:979: delete oid 46 current snap is 112 2023-12-15T23:00:30.513 INFO:tasks.rados.rados.0.smithi071.stdout:979: done (0 left) 2023-12-15T23:00:30.513 INFO:tasks.rados.rados.0.smithi071.stdout:980: write (excl) oid 37 current snap is 112 2023-12-15T23:00:30.513 INFO:tasks.rados.rados.0.smithi071.stdout:980: seq_num 478 ranges {433746=782908,1954981=78765} 2023-12-15T23:00:30.520 INFO:tasks.rados.rados.0.smithi071.stdout:980: writing smithi07183189-37 from 433746 to 1216654 tid 1 2023-12-15T23:00:30.525 INFO:tasks.rados.rados.0.smithi071.stdout:980: writing smithi07183189-37 from 1954981 to 2033746 tid 2 2023-12-15T23:00:30.525 INFO:tasks.rados.rados.0.smithi071.stdout:981: read oid 29 snap -1 2023-12-15T23:00:30.525 INFO:tasks.rados.rados.0.smithi071.stdout:981: expect deleted 2023-12-15T23:00:30.526 INFO:tasks.rados.rados.0.smithi071.stdout:982: read oid 10 snap -1 2023-12-15T23:00:30.526 INFO:tasks.rados.rados.0.smithi071.stdout:982: expect deleted 2023-12-15T23:00:30.526 INFO:tasks.rados.rados.0.smithi071.stdout:983: read oid 14 snap -1 2023-12-15T23:00:30.526 INFO:tasks.rados.rados.0.smithi071.stdout:983: expect deleted 2023-12-15T23:00:30.526 INFO:tasks.rados.rados.0.smithi071.stdout:984: write (excl) oid 34 current snap is 112 2023-12-15T23:00:30.526 INFO:tasks.rados.rados.0.smithi071.stdout:984: seq_num 479 ranges {450553=574510,1613463=437090} 2023-12-15T23:00:30.533 INFO:tasks.rados.rados.0.smithi071.stdout:984: writing smithi07183189-34 from 450553 to 1025063 tid 1 2023-12-15T23:00:30.538 INFO:tasks.rados.rados.0.smithi071.stdout:984: writing smithi07183189-34 from 1613463 to 2050553 tid 2 2023-12-15T23:00:30.538 INFO:tasks.rados.rados.0.smithi071.stdout:985: read oid 47 snap -1 2023-12-15T23:00:30.539 INFO:tasks.rados.rados.0.smithi071.stdout:985: expect (ObjNum 464 snap 108 seq_num 464) 2023-12-15T23:00:30.539 INFO:tasks.rados.rados.0.smithi071.stdout:986: write (excl) oid 20 current snap is 112 2023-12-15T23:00:30.539 INFO:tasks.rados.rados.0.smithi071.stdout:986: seq_num 480 ranges {467360=649759,1671945=395415} 2023-12-15T23:00:30.546 INFO:tasks.rados.rados.0.smithi071.stdout:986: writing smithi07183189-20 from 467360 to 1117119 tid 1 2023-12-15T23:00:30.553 INFO:tasks.rados.rados.0.smithi071.stdout:986: writing smithi07183189-20 from 1671945 to 2067360 tid 2 2023-12-15T23:00:30.553 INFO:tasks.rados.rados.0.smithi071.stdout:987: read oid 8 snap -1 2023-12-15T23:00:30.553 INFO:tasks.rados.rados.0.smithi071.stdout:987: expect (ObjNum 458 snap 105 seq_num 458) 2023-12-15T23:00:30.554 INFO:tasks.rados.rados.0.smithi071.stdout:988: write oid 45 current snap is 112 2023-12-15T23:00:30.554 INFO:tasks.rados.rados.0.smithi071.stdout:988: seq_num 481 ranges {484167=725008,1730427=353740} 2023-12-15T23:00:30.562 INFO:tasks.rados.rados.0.smithi071.stdout:988: writing smithi07183189-45 from 484167 to 1209175 tid 1 2023-12-15T23:00:30.568 INFO:tasks.rados.rados.0.smithi071.stdout:988: writing smithi07183189-45 from 1730427 to 2084167 tid 2 2023-12-15T23:00:30.568 INFO:tasks.rados.rados.0.smithi071.stdout:989: delete oid 21 current snap is 112 2023-12-15T23:00:30.604 INFO:tasks.rados.rados.0.smithi071.stdout:980: finishing write tid 1 to smithi07183189-37 2023-12-15T23:00:30.606 INFO:tasks.rados.rados.0.smithi071.stdout:984: finishing write tid 1 to smithi07183189-34 2023-12-15T23:00:30.615 INFO:tasks.rados.rados.0.smithi071.stdout:980: finishing write tid 2 to smithi07183189-37 2023-12-15T23:00:30.624 INFO:tasks.rados.rados.0.smithi071.stdout:980: finishing write tid 3 to smithi07183189-37 2023-12-15T23:00:30.627 INFO:tasks.rados.rados.0.smithi071.stdout:980: finishing write tid 4 to smithi07183189-37 2023-12-15T23:00:30.627 INFO:tasks.rados.rados.0.smithi071.stdout:980: oid 37 updating version 0 to 595 2023-12-15T23:00:30.627 INFO:tasks.rados.rados.0.smithi071.stdout:980: oid 37 updating version 595 to 597 2023-12-15T23:00:30.627 INFO:tasks.rados.rados.0.smithi071.stdout:980: oid 37 version 597 is already newer than 596 2023-12-15T23:00:30.627 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 597 (ObjNum 478 snap 112 seq_num 478) dirty exists 2023-12-15T23:00:30.627 INFO:tasks.rados.rados.0.smithi071.stdout:980: left oid 37 (ObjNum 478 snap 112 seq_num 478) 2023-12-15T23:00:30.629 INFO:tasks.rados.rados.0.smithi071.stdout:985: expect (ObjNum 464 snap 108 seq_num 464) 2023-12-15T23:00:30.835 INFO:tasks.rados.rados.0.smithi071.stdout:986: finishing write tid 1 to smithi07183189-20 2023-12-15T23:00:30.835 INFO:tasks.rados.rados.0.smithi071.stdout:986: finishing write tid 2 to smithi07183189-20 2023-12-15T23:00:30.835 INFO:tasks.rados.rados.0.smithi071.stdout:986: finishing write tid 3 to smithi07183189-20 2023-12-15T23:00:30.835 INFO:tasks.rados.rados.0.smithi071.stdout:986: finishing write tid 4 to smithi07183189-20 2023-12-15T23:00:30.836 INFO:tasks.rados.rados.0.smithi071.stdout:986: oid 20 updating version 0 to 598 2023-12-15T23:00:30.836 INFO:tasks.rados.rados.0.smithi071.stdout:986: oid 20 updating version 598 to 600 2023-12-15T23:00:30.836 INFO:tasks.rados.rados.0.smithi071.stdout:986: oid 20 version 600 is already newer than 600 2023-12-15T23:00:30.836 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 20 v 600 (ObjNum 480 snap 112 seq_num 480) dirty exists 2023-12-15T23:00:30.836 INFO:tasks.rados.rados.0.smithi071.stdout:986: left oid 20 (ObjNum 480 snap 112 seq_num 480) 2023-12-15T23:00:30.836 INFO:tasks.rados.rados.0.smithi071.stdout:984: finishing write tid 2 to smithi07183189-34 2023-12-15T23:00:30.836 INFO:tasks.rados.rados.0.smithi071.stdout:988: finishing write tid 1 to smithi07183189-45 2023-12-15T23:00:30.836 INFO:tasks.rados.rados.0.smithi071.stdout:980: done (9 left) 2023-12-15T23:00:30.836 INFO:tasks.rados.rados.0.smithi071.stdout:981: done (8 left) 2023-12-15T23:00:30.836 INFO:tasks.rados.rados.0.smithi071.stdout:982: done (7 left) 2023-12-15T23:00:30.836 INFO:tasks.rados.rados.0.smithi071.stdout:983: done (6 left) 2023-12-15T23:00:30.836 INFO:tasks.rados.rados.0.smithi071.stdout:985: done (5 left) 2023-12-15T23:00:30.836 INFO:tasks.rados.rados.0.smithi071.stdout:986: done (4 left) 2023-12-15T23:00:30.836 INFO:tasks.rados.rados.0.smithi071.stdout:989: done (3 left) 2023-12-15T23:00:30.836 INFO:tasks.rados.rados.0.smithi071.stdout:990: write oid 11 current snap is 112 2023-12-15T23:00:30.836 INFO:tasks.rados.rados.0.smithi071.stdout:988: finishing write tid 2 to smithi07183189-45 2023-12-15T23:00:30.837 INFO:tasks.rados.rados.0.smithi071.stdout:987: expect (ObjNum 458 snap 105 seq_num 458) 2023-12-15T23:00:31.063 INFO:tasks.rados.rados.0.smithi071.stdout:984: finishing write tid 3 to smithi07183189-34 2023-12-15T23:00:31.063 INFO:tasks.rados.rados.0.smithi071.stdout:984: finishing write tid 4 to smithi07183189-34 2023-12-15T23:00:31.064 INFO:tasks.rados.rados.0.smithi071.stdout:984: oid 34 updating version 0 to 74 2023-12-15T23:00:31.064 INFO:tasks.rados.rados.0.smithi071.stdout:984: oid 34 updating version 74 to 76 2023-12-15T23:00:31.064 INFO:tasks.rados.rados.0.smithi071.stdout:984: oid 34 version 76 is already newer than 75 2023-12-15T23:00:31.064 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 76 (ObjNum 479 snap 112 seq_num 479) dirty exists 2023-12-15T23:00:31.064 INFO:tasks.rados.rados.0.smithi071.stdout:984: left oid 34 (ObjNum 479 snap 112 seq_num 479) 2023-12-15T23:00:31.064 INFO:tasks.rados.rados.0.smithi071.stdout:988: finishing write tid 3 to smithi07183189-45 2023-12-15T23:00:31.064 INFO:tasks.rados.rados.0.smithi071.stdout:990: seq_num 483 ranges {517781=475506,1731038=386743} 2023-12-15T23:00:31.068 INFO:tasks.rados.rados.0.smithi071.stdout:990: writing smithi07183189-11 from 517781 to 993287 tid 1 2023-12-15T23:00:31.076 INFO:tasks.rados.rados.0.smithi071.stdout:990: writing smithi07183189-11 from 1731038 to 2117781 tid 2 2023-12-15T23:00:31.076 INFO:tasks.rados.rados.0.smithi071.stdout:984: done (3 left) 2023-12-15T23:00:31.076 INFO:tasks.rados.rados.0.smithi071.stdout:987: done (2 left) 2023-12-15T23:00:31.076 INFO:tasks.rados.rados.0.smithi071.stdout:991: delete oid 36 current snap is 112 2023-12-15T23:00:31.076 INFO:tasks.rados.rados.0.smithi071.stdout:988: finishing write tid 4 to smithi07183189-45 2023-12-15T23:00:31.076 INFO:tasks.rados.rados.0.smithi071.stdout:988: oid 45 updating version 0 to 603 2023-12-15T23:00:31.077 INFO:tasks.rados.rados.0.smithi071.stdout:988: oid 45 version 603 is already newer than 602 2023-12-15T23:00:31.077 INFO:tasks.rados.rados.0.smithi071.stdout:988: oid 45 version 603 is already newer than 601 2023-12-15T23:00:31.077 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 603 (ObjNum 481 snap 112 seq_num 481) dirty exists 2023-12-15T23:00:31.077 INFO:tasks.rados.rados.0.smithi071.stdout:988: left oid 45 (ObjNum 481 snap 112 seq_num 481) 2023-12-15T23:00:31.110 INFO:tasks.rados.rados.0.smithi071.stdout:990: finishing write tid 1 to smithi07183189-11 2023-12-15T23:00:31.112 INFO:tasks.rados.rados.0.smithi071.stdout:988: done (2 left) 2023-12-15T23:00:31.112 INFO:tasks.rados.rados.0.smithi071.stdout:991: done (1 left) 2023-12-15T23:00:31.112 INFO:tasks.rados.rados.0.smithi071.stdout:992: snap_create 2023-12-15T23:00:31.116 INFO:tasks.rados.rados.0.smithi071.stdout:990: finishing write tid 2 to smithi07183189-11 2023-12-15T23:00:31.126 INFO:tasks.rados.rados.0.smithi071.stdout:990: finishing write tid 3 to smithi07183189-11 2023-12-15T23:00:31.127 INFO:tasks.rados.rados.0.smithi071.stdout:990: finishing write tid 4 to smithi07183189-11 2023-12-15T23:00:31.128 INFO:tasks.rados.rados.0.smithi071.stdout:990: oid 11 updating version 0 to 230 2023-12-15T23:00:31.128 INFO:tasks.rados.rados.0.smithi071.stdout:990: oid 11 version 230 is already newer than 229 2023-12-15T23:00:31.128 INFO:tasks.rados.rados.0.smithi071.stdout:990: oid 11 version 230 is already newer than 228 2023-12-15T23:00:31.128 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 11 v 230 (ObjNum 483 snap 112 seq_num 483) dirty exists 2023-12-15T23:00:31.128 INFO:tasks.rados.rados.0.smithi071.stdout:990: left oid 11 (ObjNum 483 snap 112 seq_num 483) 2023-12-15T23:00:31.436 INFO:tasks.rados.rados.0.smithi071.stdout:990: done (1 left) 2023-12-15T23:00:31.436 INFO:tasks.rados.rados.0.smithi071.stdout:992: done (0 left) 2023-12-15T23:00:31.436 INFO:tasks.rados.rados.0.smithi071.stdout:993: write (excl) oid 41 current snap is 113 2023-12-15T23:00:31.436 INFO:tasks.rados.rados.0.smithi071.stdout:993: seq_num 485 ranges {551395=626004,1848002=303393} 2023-12-15T23:00:31.443 INFO:tasks.rados.rados.0.smithi071.stdout:993: writing smithi07183189-41 from 551395 to 1177399 tid 1 2023-12-15T23:00:31.451 INFO:tasks.rados.rados.0.smithi071.stdout:993: writing smithi07183189-41 from 1848002 to 2151395 tid 2 2023-12-15T23:00:31.451 INFO:tasks.rados.rados.0.smithi071.stdout:994: read oid 17 snap -1 2023-12-15T23:00:31.451 INFO:tasks.rados.rados.0.smithi071.stdout:994: expect deleted 2023-12-15T23:00:31.451 INFO:tasks.rados.rados.0.smithi071.stdout:995: read oid 39 snap -1 2023-12-15T23:00:31.451 INFO:tasks.rados.rados.0.smithi071.stdout:995: expect deleted 2023-12-15T23:00:31.451 INFO:tasks.rados.rados.0.smithi071.stdout:996: read oid 18 snap -1 2023-12-15T23:00:31.451 INFO:tasks.rados.rados.0.smithi071.stdout:996: expect (ObjNum 471 snap 110 seq_num 471) 2023-12-15T23:00:31.451 INFO:tasks.rados.rados.0.smithi071.stdout:997: read oid 6 snap 112 2023-12-15T23:00:31.451 INFO:tasks.rados.rados.0.smithi071.stdout:997: expect (ObjNum 375 snap 91 seq_num 375) 2023-12-15T23:00:31.452 INFO:tasks.rados.rados.0.smithi071.stdout:998: read oid 30 snap 112 2023-12-15T23:00:31.452 INFO:tasks.rados.rados.0.smithi071.stdout:998: expect (ObjNum 448 snap 103 seq_num 448) 2023-12-15T23:00:31.452 INFO:tasks.rados.rados.0.smithi071.stdout:999: read oid 1 snap -1 2023-12-15T23:00:31.452 INFO:tasks.rados.rados.0.smithi071.stdout:999: expect (ObjNum 407 snap 98 seq_num 407) 2023-12-15T23:00:31.452 INFO:tasks.rados.rados.0.smithi071.stdout:1000: read oid 23 snap -1 2023-12-15T23:00:31.452 INFO:tasks.rados.rados.0.smithi071.stdout:1000: expect deleted 2023-12-15T23:00:31.452 INFO:tasks.rados.rados.0.smithi071.stdout:1001: write (excl) oid 34 current snap is 113 2023-12-15T23:00:31.452 INFO:tasks.rados.rados.0.smithi071.stdout:1001: seq_num 486 ranges {568202=701253,1906484=261718} 2023-12-15T23:00:31.453 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T23:00:31.453 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T23:00:31.454 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-12-15T23:00:31.454 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-12-15T23:00:31.458 INFO:tasks.rados.rados.0.smithi071.stdout:1001: writing smithi07183189-34 from 568202 to 1269455 tid 1 2023-12-15T23:00:31.463 INFO:tasks.rados.rados.0.smithi071.stdout:1001: writing smithi07183189-34 from 1906484 to 2168202 tid 2 2023-12-15T23:00:31.463 INFO:tasks.rados.rados.0.smithi071.stdout:1002: write oid 45 current snap is 113 2023-12-15T23:00:31.464 INFO:tasks.rados.rados.0.smithi071.stdout:1002: seq_num 487 ranges {585009=492855,1564966=620043} 2023-12-15T23:00:31.470 INFO:tasks.rados.rados.0.smithi071.stdout:1002: writing smithi07183189-45 from 585009 to 1077864 tid 1 2023-12-15T23:00:31.478 INFO:tasks.rados.rados.0.smithi071.stdout:1002: writing smithi07183189-45 from 1564966 to 2185009 tid 2 2023-12-15T23:00:31.478 INFO:tasks.rados.rados.0.smithi071.stdout:1003: read oid 4 snap -1 2023-12-15T23:00:31.478 INFO:tasks.rados.rados.0.smithi071.stdout:1003: expect (ObjNum 422 snap 99 seq_num 422) 2023-12-15T23:00:31.478 INFO:tasks.rados.rados.0.smithi071.stdout:1004: read oid 44 snap -1 2023-12-15T23:00:31.478 INFO:tasks.rados.rados.0.smithi071.stdout:1004: expect (ObjNum 467 snap 110 seq_num 467) 2023-12-15T23:00:31.478 INFO:tasks.rados.rados.0.smithi071.stdout:1005: delete oid 36 current snap is 113 2023-12-15T23:00:31.493 INFO:tasks.rados.rados.0.smithi071.stdout:997: expect (ObjNum 375 snap 91 seq_num 375) 2023-12-15T23:00:31.580 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:31.575+0000 7f9dd075e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:31.583 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:31.583+0000 7f9dd075e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:31.690 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T23:00:31.690 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T23:00:31.690 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T23:00:31.690 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T23:00:31.691 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T23:00:31.691 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T23:00:31.691 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T23:00:31.691 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T23:00:31.691 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T23:00:31.691 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T23:00:31.691 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T23:00:31.691 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T23:00:31.693 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T23:00:31.693 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:00:31.693 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-12-15T23:00:31.761 INFO:tasks.rados.rados.0.smithi071.stdout:994: done (12 left) 2023-12-15T23:00:31.761 INFO:tasks.rados.rados.0.smithi071.stdout:995: done (11 left) 2023-12-15T23:00:31.761 INFO:tasks.rados.rados.0.smithi071.stdout:997: done (10 left) 2023-12-15T23:00:31.761 INFO:tasks.rados.rados.0.smithi071.stdout:1005: done (9 left) 2023-12-15T23:00:31.761 INFO:tasks.rados.rados.0.smithi071.stdout:1006: write (excl) oid 38 current snap is 113 2023-12-15T23:00:31.761 INFO:tasks.rados.rados.0.smithi071.stdout:1004: expect (ObjNum 467 snap 110 seq_num 467) 2023-12-15T23:00:31.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:31.835+0000 7ff9fcd9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:31.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:31.842+0000 7ff9fcd9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:31.857 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T23:00:31.858 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T23:00:31.858 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T23:00:31.858 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T23:00:31.858 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T23:00:31.858 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T23:00:31.858 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T23:00:31.858 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T23:00:31.858 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T23:00:31.858 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T23:00:31.858 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T23:00:31.858 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T23:00:31.860 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T23:00:31.861 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:00:31.861 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-12-15T23:00:32.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:32.003+0000 7ff92d06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:32.010 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:32.010+0000 7ff92d06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:32.023 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T23:00:32.023 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T23:00:32.023 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T23:00:32.023 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T23:00:32.023 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T23:00:32.023 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T23:00:32.023 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T23:00:32.023 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T23:00:32.023 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T23:00:32.023 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T23:00:32.023 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T23:00:32.023 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T23:00:32.026 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T23:00:32.026 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:00:32.026 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-12-15T23:00:32.047 INFO:tasks.rados.rados.0.smithi071.stdout:998: expect (ObjNum 448 snap 103 seq_num 448) 2023-12-15T23:00:32.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:32.128+0000 7fb432b04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:32.136 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:32.136+0000 7fb432b04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:32.153 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T23:00:32.153 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T23:00:32.153 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T23:00:32.153 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T23:00:32.153 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T23:00:32.153 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T23:00:32.153 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T23:00:32.154 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T23:00:32.154 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T23:00:32.154 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T23:00:32.154 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T23:00:32.154 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T23:00:32.157 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T23:00:32.157 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:00:32.245 INFO:tasks.rados.rados.0.smithi071.stdout:996: expect (ObjNum 471 snap 110 seq_num 471) 2023-12-15T23:00:32.366 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T23:00:32.402 INFO:tasks.rados.rados.0.smithi071.stdout:999: expect (ObjNum 407 snap 98 seq_num 407) 2023-12-15T23:00:32.469 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:32.468+0000 7fbeb7b1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:32.472 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:32.472+0000 7fbeb7b1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:32.569 INFO:tasks.rados.rados.0.smithi071.stdout:1003: expect (ObjNum 422 snap 99 seq_num 422) 2023-12-15T23:00:32.776 INFO:tasks.rados.rados.0.smithi071.stdout:1001: finishing write tid 1 to smithi07183189-34 2023-12-15T23:00:32.777 INFO:tasks.rados.rados.0.smithi071.stdout:993: finishing write tid 1 to smithi07183189-41 2023-12-15T23:00:32.777 INFO:tasks.rados.rados.0.smithi071.stdout:1002: finishing write tid 1 to smithi07183189-45 2023-12-15T23:00:32.777 INFO:tasks.rados.rados.0.smithi071.stdout:1001: finishing write tid 2 to smithi07183189-34 2023-12-15T23:00:32.777 INFO:tasks.rados.rados.0.smithi071.stdout:1002: finishing write tid 2 to smithi07183189-45 2023-12-15T23:00:32.777 INFO:tasks.rados.rados.0.smithi071.stdout:993: finishing write tid 2 to smithi07183189-41 2023-12-15T23:00:32.777 INFO:tasks.rados.rados.0.smithi071.stdout:1001: finishing write tid 3 to smithi07183189-34 2023-12-15T23:00:32.777 INFO:tasks.rados.rados.0.smithi071.stdout:1006: seq_num 489 ranges {618623=643353,1681930=536693} 2023-12-15T23:00:32.783 INFO:tasks.rados.rados.0.smithi071.stdout:1006: writing smithi07183189-38 from 618623 to 1261976 tid 1 2023-12-15T23:00:32.789 INFO:tasks.rados.rados.0.smithi071.stdout:1006: writing smithi07183189-38 from 1681930 to 2218623 tid 2 2023-12-15T23:00:32.790 INFO:tasks.rados.rados.0.smithi071.stdout:996: done (9 left) 2023-12-15T23:00:32.790 INFO:tasks.rados.rados.0.smithi071.stdout:998: done (8 left) 2023-12-15T23:00:32.790 INFO:tasks.rados.rados.0.smithi071.stdout:999: done (7 left) 2023-12-15T23:00:32.790 INFO:tasks.rados.rados.0.smithi071.stdout:1000: done (6 left) 2023-12-15T23:00:32.790 INFO:tasks.rados.rados.0.smithi071.stdout:1003: done (5 left) 2023-12-15T23:00:32.790 INFO:tasks.rados.rados.0.smithi071.stdout:1004: done (4 left) 2023-12-15T23:00:32.790 INFO:tasks.rados.rados.0.smithi071.stdout:1007: read oid 14 snap 112 2023-12-15T23:00:32.790 INFO:tasks.rados.rados.0.smithi071.stdout:1007: expect deleted 2023-12-15T23:00:32.790 INFO:tasks.rados.rados.0.smithi071.stdout:1008: delete oid 15 current snap is 113 2023-12-15T23:00:32.790 INFO:tasks.rados.rados.0.smithi071.stdout:1001: finishing write tid 4 to smithi07183189-34 2023-12-15T23:00:32.790 INFO:tasks.rados.rados.0.smithi071.stdout:1001: oid 34 updating version 0 to 78 2023-12-15T23:00:32.791 INFO:tasks.rados.rados.0.smithi071.stdout:1001: oid 34 updating version 78 to 80 2023-12-15T23:00:32.791 INFO:tasks.rados.rados.0.smithi071.stdout:1001: oid 34 version 80 is already newer than 79 2023-12-15T23:00:32.791 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 80 (ObjNum 486 snap 113 seq_num 486) dirty exists 2023-12-15T23:00:32.791 INFO:tasks.rados.rados.0.smithi071.stdout:1001: left oid 34 (ObjNum 486 snap 113 seq_num 486) 2023-12-15T23:00:32.791 INFO:tasks.rados.rados.0.smithi071.stdout:1002: finishing write tid 3 to smithi07183189-45 2023-12-15T23:00:32.791 INFO:tasks.rados.rados.0.smithi071.stdout:1002: finishing write tid 4 to smithi07183189-45 2023-12-15T23:00:32.791 INFO:tasks.rados.rados.0.smithi071.stdout:1002: oid 45 updating version 0 to 607 2023-12-15T23:00:32.791 INFO:tasks.rados.rados.0.smithi071.stdout:1002: oid 45 version 607 is already newer than 606 2023-12-15T23:00:32.791 INFO:tasks.rados.rados.0.smithi071.stdout:1002: oid 45 version 607 is already newer than 606 2023-12-15T23:00:32.791 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 607 (ObjNum 487 snap 113 seq_num 487) dirty exists 2023-12-15T23:00:32.791 INFO:tasks.rados.rados.0.smithi071.stdout:1002: left oid 45 (ObjNum 487 snap 113 seq_num 487) 2023-12-15T23:00:32.791 INFO:tasks.rados.rados.0.smithi071.stdout:993: finishing write tid 3 to smithi07183189-41 2023-12-15T23:00:32.791 INFO:tasks.rados.rados.0.smithi071.stdout:993: finishing write tid 4 to smithi07183189-41 2023-12-15T23:00:32.792 INFO:tasks.rados.rados.0.smithi071.stdout:993: oid 41 updating version 0 to 294 2023-12-15T23:00:32.792 INFO:tasks.rados.rados.0.smithi071.stdout:993: oid 41 version 294 is already newer than 292 2023-12-15T23:00:32.792 INFO:tasks.rados.rados.0.smithi071.stdout:993: oid 41 version 294 is already newer than 293 2023-12-15T23:00:32.792 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 41 v 294 (ObjNum 485 snap 113 seq_num 485) dirty exists 2023-12-15T23:00:32.792 INFO:tasks.rados.rados.0.smithi071.stdout:993: left oid 41 (ObjNum 485 snap 113 seq_num 485) 2023-12-15T23:00:32.812 INFO:tasks.rados.rados.0.smithi071.stdout:1006: finishing write tid 1 to smithi07183189-38 2023-12-15T23:00:32.824 INFO:tasks.rados.rados.0.smithi071.stdout:993: done (5 left) 2023-12-15T23:00:32.824 INFO:tasks.rados.rados.0.smithi071.stdout:1001: done (4 left) 2023-12-15T23:00:32.824 INFO:tasks.rados.rados.0.smithi071.stdout:1002: done (3 left) 2023-12-15T23:00:32.824 INFO:tasks.rados.rados.0.smithi071.stdout:1007: done (2 left) 2023-12-15T23:00:32.824 INFO:tasks.rados.rados.0.smithi071.stdout:1008: done (1 left) 2023-12-15T23:00:32.824 INFO:tasks.rados.rados.0.smithi071.stdout:1009: snap_remove snap 112 2023-12-15T23:00:33.441 INFO:tasks.rados.rados.0.smithi071.stdout:1009: done (1 left) 2023-12-15T23:00:33.441 INFO:tasks.rados.rados.0.smithi071.stdout:1010: write oid 3 current snap is 113 2023-12-15T23:00:33.441 INFO:tasks.rados.rados.0.smithi071.stdout:1006: finishing write tid 2 to smithi07183189-38 2023-12-15T23:00:33.441 INFO:tasks.rados.rados.0.smithi071.stdout:1010: seq_num 491 ranges {652237=793851,2198894=53343} 2023-12-15T23:00:33.448 INFO:tasks.rados.rados.0.smithi071.stdout:1010: writing smithi07183189-3 from 652237 to 1446088 tid 1 2023-12-15T23:00:33.453 INFO:tasks.rados.rados.0.smithi071.stdout:1010: writing smithi07183189-3 from 2198894 to 2252237 tid 2 2023-12-15T23:00:33.453 INFO:tasks.rados.rados.0.smithi071.stdout:1011: write (excl) oid 11 current snap is 113 2023-12-15T23:00:33.454 INFO:tasks.rados.rados.0.smithi071.stdout:1011: seq_num 492 ranges {669044=469100,1741023=459161} 2023-12-15T23:00:33.459 INFO:tasks.rados.rados.0.smithi071.stdout:1011: writing smithi07183189-11 from 669044 to 1138144 tid 1 2023-12-15T23:00:33.466 INFO:tasks.rados.rados.0.smithi071.stdout:1011: writing smithi07183189-11 from 1741023 to 2200184 tid 2 2023-12-15T23:00:33.466 INFO:tasks.rados.rados.0.smithi071.stdout:1012: read oid 50 snap -1 2023-12-15T23:00:33.466 INFO:tasks.rados.rados.0.smithi071.stdout:1012: expect deleted 2023-12-15T23:00:33.466 INFO:tasks.rados.rados.0.smithi071.stdout:1013: write oid 27 current snap is 113 2023-12-15T23:00:33.466 INFO:tasks.rados.rados.0.smithi071.stdout:1006: finishing write tid 3 to smithi07183189-38 2023-12-15T23:00:33.466 INFO:tasks.rados.rados.0.smithi071.stdout:1013: seq_num 493 ranges {685851=544349,1799505=486346} 2023-12-15T23:00:33.473 INFO:tasks.rados.rados.0.smithi071.stdout:1013: writing smithi07183189-27 from 685851 to 1230200 tid 1 2023-12-15T23:00:33.479 INFO:tasks.rados.rados.0.smithi071.stdout:1013: writing smithi07183189-27 from 1799505 to 2285851 tid 2 2023-12-15T23:00:33.480 INFO:tasks.rados.rados.0.smithi071.stdout:1014: write oid 43 current snap is 113 2023-12-15T23:00:33.480 INFO:tasks.rados.rados.0.smithi071.stdout:1006: finishing write tid 4 to smithi07183189-38 2023-12-15T23:00:33.480 INFO:tasks.rados.rados.0.smithi071.stdout:1006: oid 38 updating version 0 to 152 2023-12-15T23:00:33.480 INFO:tasks.rados.rados.0.smithi071.stdout:1006: oid 38 version 152 is already newer than 151 2023-12-15T23:00:33.480 INFO:tasks.rados.rados.0.smithi071.stdout:1006: oid 38 version 152 is already newer than 150 2023-12-15T23:00:33.480 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 38 v 152 (ObjNum 489 snap 113 seq_num 489) dirty exists 2023-12-15T23:00:33.480 INFO:tasks.rados.rados.0.smithi071.stdout:1006: left oid 38 (ObjNum 489 snap 113 seq_num 489) 2023-12-15T23:00:33.480 INFO:tasks.rados.rados.0.smithi071.stdout:1014: seq_num 494 ranges {702658=619598,1857987=444671} 2023-12-15T23:00:33.486 INFO:tasks.rados.rados.0.smithi071.stdout:1014: writing smithi07183189-43 from 702658 to 1322256 tid 1 2023-12-15T23:00:33.492 INFO:tasks.rados.rados.0.smithi071.stdout:1014: writing smithi07183189-43 from 1857987 to 2302658 tid 2 2023-12-15T23:00:33.492 INFO:tasks.rados.rados.0.smithi071.stdout:1006: done (5 left) 2023-12-15T23:00:33.492 INFO:tasks.rados.rados.0.smithi071.stdout:1015: delete oid 7 current snap is 113 2023-12-15T23:00:33.567 INFO:tasks.rados.rados.0.smithi071.stdout:1011: finishing write tid 1 to smithi07183189-11 2023-12-15T23:00:33.584 INFO:tasks.rados.rados.0.smithi071.stdout:1013: finishing write tid 1 to smithi07183189-27 2023-12-15T23:00:33.588 INFO:tasks.rados.rados.0.smithi071.stdout:1010: finishing write tid 1 to smithi07183189-3 2023-12-15T23:00:33.588 INFO:tasks.rados.rados.0.smithi071.stdout:1012: done (5 left) 2023-12-15T23:00:33.588 INFO:tasks.rados.rados.0.smithi071.stdout:1015: done (4 left) 2023-12-15T23:00:33.588 INFO:tasks.rados.rados.0.smithi071.stdout:1016: delete oid 34 current snap is 113 2023-12-15T23:00:33.590 INFO:tasks.rados.rados.0.smithi071.stdout:1014: finishing write tid 1 to smithi07183189-43 2023-12-15T23:00:33.607 INFO:tasks.rados.rados.0.smithi071.stdout:1011: finishing write tid 2 to smithi07183189-11 2023-12-15T23:00:33.625 INFO:tasks.rados.rados.0.smithi071.stdout:1011: finishing write tid 3 to smithi07183189-11 2023-12-15T23:00:33.628 INFO:tasks.rados.rados.0.smithi071.stdout:1011: finishing write tid 4 to smithi07183189-11 2023-12-15T23:00:33.628 INFO:tasks.rados.rados.0.smithi071.stdout:1011: oid 11 updating version 0 to 235 2023-12-15T23:00:33.628 INFO:tasks.rados.rados.0.smithi071.stdout:1011: oid 11 updating version 235 to 236 2023-12-15T23:00:33.628 INFO:tasks.rados.rados.0.smithi071.stdout:1011: oid 11 updating version 236 to 237 2023-12-15T23:00:33.628 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 11 v 237 (ObjNum 492 snap 113 seq_num 492) dirty exists 2023-12-15T23:00:33.628 INFO:tasks.rados.rados.0.smithi071.stdout:1011: left oid 11 (ObjNum 492 snap 113 seq_num 492) 2023-12-15T23:00:33.628 INFO:tasks.rados.rados.0.smithi071.stdout:1010: finishing write tid 2 to smithi07183189-3 2023-12-15T23:00:33.628 INFO:tasks.rados.rados.0.smithi071.stdout:1013: finishing write tid 2 to smithi07183189-27 2023-12-15T23:00:33.634 INFO:tasks.rados.rados.0.smithi071.stdout:1014: finishing write tid 2 to smithi07183189-43 2023-12-15T23:00:33.652 INFO:tasks.rados.rados.0.smithi071.stdout:1011: done (4 left) 2023-12-15T23:00:33.652 INFO:tasks.rados.rados.0.smithi071.stdout:1016: done (3 left) 2023-12-15T23:00:33.652 INFO:tasks.rados.rados.0.smithi071.stdout:1017: delete oid 39 current snap is 113 2023-12-15T23:00:33.658 INFO:tasks.rados.rados.0.smithi071.stdout:1010: finishing write tid 3 to smithi07183189-3 2023-12-15T23:00:33.658 INFO:tasks.rados.rados.0.smithi071.stdout:1010: finishing write tid 4 to smithi07183189-3 2023-12-15T23:00:33.658 INFO:tasks.rados.rados.0.smithi071.stdout:1010: oid 3 updating version 0 to 154 2023-12-15T23:00:33.658 INFO:tasks.rados.rados.0.smithi071.stdout:1010: oid 3 version 154 is already newer than 153 2023-12-15T23:00:33.658 INFO:tasks.rados.rados.0.smithi071.stdout:1010: oid 3 updating version 154 to 155 2023-12-15T23:00:33.659 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 155 (ObjNum 491 snap 113 seq_num 491) dirty exists 2023-12-15T23:00:33.659 INFO:tasks.rados.rados.0.smithi071.stdout:1010: left oid 3 (ObjNum 491 snap 113 seq_num 491) 2023-12-15T23:00:33.670 INFO:tasks.rados.rados.0.smithi071.stdout:1013: finishing write tid 3 to smithi07183189-27 2023-12-15T23:00:33.671 INFO:tasks.rados.rados.0.smithi071.stdout:1014: finishing write tid 3 to smithi07183189-43 2023-12-15T23:00:33.672 INFO:tasks.rados.rados.0.smithi071.stdout:1014: finishing write tid 4 to smithi07183189-43 2023-12-15T23:00:33.672 INFO:tasks.rados.rados.0.smithi071.stdout:1014: oid 43 updating version 0 to 126 2023-12-15T23:00:33.672 INFO:tasks.rados.rados.0.smithi071.stdout:1014: oid 43 version 126 is already newer than 125 2023-12-15T23:00:33.672 INFO:tasks.rados.rados.0.smithi071.stdout:1014: oid 43 updating version 126 to 127 2023-12-15T23:00:33.672 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 127 (ObjNum 494 snap 113 seq_num 494) dirty exists 2023-12-15T23:00:33.672 INFO:tasks.rados.rados.0.smithi071.stdout:1014: left oid 43 (ObjNum 494 snap 113 seq_num 494) 2023-12-15T23:00:33.673 INFO:tasks.rados.rados.0.smithi071.stdout:1013: finishing write tid 4 to smithi07183189-27 2023-12-15T23:00:33.673 INFO:tasks.rados.rados.0.smithi071.stdout:1013: oid 27 updating version 0 to 239 2023-12-15T23:00:33.673 INFO:tasks.rados.rados.0.smithi071.stdout:1013: oid 27 version 239 is already newer than 238 2023-12-15T23:00:33.673 INFO:tasks.rados.rados.0.smithi071.stdout:1013: oid 27 updating version 239 to 240 2023-12-15T23:00:33.673 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 27 v 240 (ObjNum 493 snap 113 seq_num 493) dirty exists 2023-12-15T23:00:33.673 INFO:tasks.rados.rados.0.smithi071.stdout:1013: left oid 27 (ObjNum 493 snap 113 seq_num 493) 2023-12-15T23:00:33.680 INFO:tasks.rados.rados.0.smithi071.stdout:1010: done (3 left) 2023-12-15T23:00:33.680 INFO:tasks.rados.rados.0.smithi071.stdout:1013: done (2 left) 2023-12-15T23:00:33.680 INFO:tasks.rados.rados.0.smithi071.stdout:1014: done (1 left) 2023-12-15T23:00:33.680 INFO:tasks.rados.rados.0.smithi071.stdout:1017: done (0 left) 2023-12-15T23:00:33.680 INFO:tasks.rados.rados.0.smithi071.stdout:1018: delete oid 47 current snap is 113 2023-12-15T23:00:33.699 INFO:tasks.rados.rados.0.smithi071.stdout:1018: done (0 left) 2023-12-15T23:00:33.699 INFO:tasks.rados.rados.0.smithi071.stdout:1019: delete oid 44 current snap is 113 2023-12-15T23:00:33.722 INFO:tasks.rados.rados.0.smithi071.stdout:1019: done (0 left) 2023-12-15T23:00:33.722 INFO:tasks.rados.rados.0.smithi071.stdout:1020: write (excl) oid 14 current snap is 113 2023-12-15T23:00:33.722 INFO:tasks.rados.rados.0.smithi071.stdout:1020: seq_num 500 ranges {403500=787445,1692526=473837} 2023-12-15T23:00:33.728 INFO:tasks.rados.rados.0.smithi071.stdout:1020: writing smithi07183189-14 from 403500 to 1190945 tid 1 2023-12-15T23:00:33.734 INFO:tasks.rados.rados.0.smithi071.stdout:1020: writing smithi07183189-14 from 1692526 to 2166363 tid 2 2023-12-15T23:00:33.734 INFO:tasks.rados.rados.0.smithi071.stdout:1021: snap_create 2023-12-15T23:00:33.747 INFO:tasks.rados.rados.0.smithi071.stdout:1020: finishing write tid 1 to smithi07183189-14 2023-12-15T23:00:33.762 INFO:tasks.rados.rados.0.smithi071.stdout:1020: finishing write tid 2 to smithi07183189-14 2023-12-15T23:00:33.771 INFO:tasks.rados.rados.0.smithi071.stdout:1020: finishing write tid 3 to smithi07183189-14 2023-12-15T23:00:33.772 INFO:tasks.rados.rados.0.smithi071.stdout:1020: finishing write tid 4 to smithi07183189-14 2023-12-15T23:00:33.772 INFO:tasks.rados.rados.0.smithi071.stdout:1020: oid 14 updating version 0 to 297 2023-12-15T23:00:33.772 INFO:tasks.rados.rados.0.smithi071.stdout:1020: oid 14 updating version 297 to 298 2023-12-15T23:00:33.772 INFO:tasks.rados.rados.0.smithi071.stdout:1020: oid 14 updating version 298 to 299 2023-12-15T23:00:33.772 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 14 v 299 (ObjNum 500 snap 113 seq_num 500) dirty exists 2023-12-15T23:00:33.772 INFO:tasks.rados.rados.0.smithi071.stdout:1020: left oid 14 (ObjNum 500 snap 113 seq_num 500) 2023-12-15T23:00:34.446 INFO:tasks.rados.rados.0.smithi071.stdout:1020: done (1 left) 2023-12-15T23:00:34.446 INFO:tasks.rados.rados.0.smithi071.stdout:1021: done (0 left) 2023-12-15T23:00:34.446 INFO:tasks.rados.rados.0.smithi071.stdout:1022: delete oid 1 current snap is 114 2023-12-15T23:00:34.482 INFO:tasks.rados.rados.0.smithi071.stdout:1022: done (0 left) 2023-12-15T23:00:34.482 INFO:tasks.rados.rados.0.smithi071.stdout:1023: write oid 25 current snap is 114 2023-12-15T23:00:34.482 INFO:tasks.rados.rados.0.smithi071.stdout:1023: seq_num 502 ranges {437114=654296,1525843=477506} 2023-12-15T23:00:34.488 INFO:tasks.rados.rados.0.smithi071.stdout:1023: writing smithi07183189-25 from 437114 to 1091410 tid 1 2023-12-15T23:00:34.493 INFO:tasks.rados.rados.0.smithi071.stdout:1023: writing smithi07183189-25 from 1525843 to 2003349 tid 2 2023-12-15T23:00:34.493 INFO:tasks.rados.rados.0.smithi071.stdout:1024: write oid 37 current snap is 114 2023-12-15T23:00:34.494 INFO:tasks.rados.rados.0.smithi071.stdout:1024: seq_num 503 ranges {453921=729545,1584325=621164} 2023-12-15T23:00:34.500 INFO:tasks.rados.rados.0.smithi071.stdout:1024: writing smithi07183189-37 from 453921 to 1183466 tid 1 2023-12-15T23:00:34.505 INFO:tasks.rados.rados.0.smithi071.stdout:1024: writing smithi07183189-37 from 1584325 to 2205489 tid 2 2023-12-15T23:00:34.506 INFO:tasks.rados.rados.0.smithi071.stdout:1025: snap_create 2023-12-15T23:00:34.555 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T23:00:34.565 INFO:tasks.rados.rados.0.smithi071.stdout:1023: finishing write tid 1 to smithi07183189-25 2023-12-15T23:00:34.568 INFO:tasks.rados.rados.0.smithi071.stdout:1024: finishing write tid 1 to smithi07183189-37 2023-12-15T23:00:34.587 INFO:tasks.rados.rados.0.smithi071.stdout:1024: finishing write tid 2 to smithi07183189-37 2023-12-15T23:00:34.592 INFO:tasks.rados.rados.0.smithi071.stdout:1023: finishing write tid 2 to smithi07183189-25 2023-12-15T23:00:34.603 INFO:tasks.rados.rados.0.smithi071.stdout:1023: finishing write tid 3 to smithi07183189-25 2023-12-15T23:00:34.605 INFO:tasks.rados.rados.0.smithi071.stdout:1024: finishing write tid 3 to smithi07183189-37 2023-12-15T23:00:34.606 INFO:tasks.rados.rados.0.smithi071.stdout:1024: finishing write tid 4 to smithi07183189-37 2023-12-15T23:00:34.606 INFO:tasks.rados.rados.0.smithi071.stdout:1024: oid 37 updating version 0 to 611 2023-12-15T23:00:34.606 INFO:tasks.rados.rados.0.smithi071.stdout:1024: oid 37 updating version 611 to 612 2023-12-15T23:00:34.606 INFO:tasks.rados.rados.0.smithi071.stdout:1024: oid 37 updating version 612 to 613 2023-12-15T23:00:34.606 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 613 (ObjNum 503 snap 114 seq_num 503) dirty exists 2023-12-15T23:00:34.606 INFO:tasks.rados.rados.0.smithi071.stdout:1024: left oid 37 (ObjNum 503 snap 114 seq_num 503) 2023-12-15T23:00:34.607 INFO:tasks.rados.rados.0.smithi071.stdout:1023: finishing write tid 4 to smithi07183189-25 2023-12-15T23:00:34.607 INFO:tasks.rados.rados.0.smithi071.stdout:1023: oid 25 updating version 0 to 321 2023-12-15T23:00:34.607 INFO:tasks.rados.rados.0.smithi071.stdout:1023: oid 25 version 321 is already newer than 321 2023-12-15T23:00:34.607 INFO:tasks.rados.rados.0.smithi071.stdout:1023: oid 25 updating version 321 to 322 2023-12-15T23:00:34.607 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 25 v 322 (ObjNum 502 snap 114 seq_num 502) dirty exists 2023-12-15T23:00:34.607 INFO:tasks.rados.rados.0.smithi071.stdout:1023: left oid 25 (ObjNum 502 snap 114 seq_num 502) 2023-12-15T23:00:34.663 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:34.662+0000 7f9dffe6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:34.666 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:34.666+0000 7f9dffe6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:35.456 INFO:tasks.rados.rados.0.smithi071.stdout:1023: done (2 left) 2023-12-15T23:00:35.456 INFO:tasks.rados.rados.0.smithi071.stdout:1024: done (1 left) 2023-12-15T23:00:35.456 INFO:tasks.rados.rados.0.smithi071.stdout:1025: done (0 left) 2023-12-15T23:00:35.456 INFO:tasks.rados.rados.0.smithi071.stdout:1026: write oid 23 current snap is 115 2023-12-15T23:00:35.456 INFO:tasks.rados.rados.0.smithi071.stdout:1026: seq_num 504 ranges {470728=404794,1526454=481175} 2023-12-15T23:00:35.460 INFO:tasks.rados.rados.0.smithi071.stdout:1026: writing smithi07183189-23 from 470728 to 875522 tid 1 2023-12-15T23:00:35.467 INFO:tasks.rados.rados.0.smithi071.stdout:1026: writing smithi07183189-23 from 1526454 to 2007629 tid 2 2023-12-15T23:00:35.467 INFO:tasks.rados.rados.0.smithi071.stdout:1027: snap_create 2023-12-15T23:00:35.499 INFO:tasks.rados.rados.0.smithi071.stdout:1026: finishing write tid 1 to smithi07183189-23 2023-12-15T23:00:35.519 INFO:tasks.rados.rados.0.smithi071.stdout:1026: finishing write tid 2 to smithi07183189-23 2023-12-15T23:00:35.520 INFO:tasks.rados.rados.0.smithi071.stdout:1026: finishing write tid 3 to smithi07183189-23 2023-12-15T23:00:35.522 INFO:tasks.rados.rados.0.smithi071.stdout:1026: finishing write tid 4 to smithi07183189-23 2023-12-15T23:00:35.522 INFO:tasks.rados.rados.0.smithi071.stdout:1026: oid 23 updating version 0 to 198 2023-12-15T23:00:35.522 INFO:tasks.rados.rados.0.smithi071.stdout:1026: oid 23 version 198 is already newer than 196 2023-12-15T23:00:35.522 INFO:tasks.rados.rados.0.smithi071.stdout:1026: oid 23 version 198 is already newer than 198 2023-12-15T23:00:35.522 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 23 v 198 (ObjNum 504 snap 115 seq_num 504) dirty exists 2023-12-15T23:00:35.522 INFO:tasks.rados.rados.0.smithi071.stdout:1026: left oid 23 (ObjNum 504 snap 115 seq_num 504) 2023-12-15T23:00:36.458 INFO:tasks.rados.rados.0.smithi071.stdout:1026: done (1 left) 2023-12-15T23:00:36.458 INFO:tasks.rados.rados.0.smithi071.stdout:1027: done (0 left) 2023-12-15T23:00:36.459 INFO:tasks.rados.rados.0.smithi071.stdout:1028: read oid 17 snap -1 2023-12-15T23:00:36.459 INFO:tasks.rados.rados.0.smithi071.stdout:1028: expect deleted 2023-12-15T23:00:36.459 INFO:tasks.rados.rados.0.smithi071.stdout:1029: read oid 50 snap -1 2023-12-15T23:00:36.459 INFO:tasks.rados.rados.0.smithi071.stdout:1029: expect deleted 2023-12-15T23:00:36.459 INFO:tasks.rados.rados.0.smithi071.stdout:1030: write oid 48 current snap is 116 2023-12-15T23:00:36.459 INFO:tasks.rados.rados.0.smithi071.stdout:1030: seq_num 505 ranges {487535=480043,1584936=624833} 2023-12-15T23:00:36.463 INFO:tasks.rados.rados.0.smithi071.stdout:1030: writing smithi07183189-48 from 487535 to 967578 tid 1 2023-12-15T23:00:36.471 INFO:tasks.rados.rados.0.smithi071.stdout:1030: writing smithi07183189-48 from 1584936 to 2209769 tid 2 2023-12-15T23:00:36.471 INFO:tasks.rados.rados.0.smithi071.stdout:1031: write oid 30 current snap is 116 2023-12-15T23:00:36.471 INFO:tasks.rados.rados.0.smithi071.stdout:1031: seq_num 506 ranges {504342=555292,1643418=484844} 2023-12-15T23:00:36.477 INFO:tasks.rados.rados.0.smithi071.stdout:1031: writing smithi07183189-30 from 504342 to 1059634 tid 1 2023-12-15T23:00:36.483 INFO:tasks.rados.rados.0.smithi071.stdout:1031: writing smithi07183189-30 from 1643418 to 2128262 tid 2 2023-12-15T23:00:36.483 INFO:tasks.rados.rados.0.smithi071.stdout:1032: snap_remove snap 114 2023-12-15T23:00:36.733 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T23:00:36.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:36.836+0000 7f646e15e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:36.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:36.840+0000 7f646e15e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:37.157 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T23:00:37.158 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T23:00:37.158 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-12-15T23:00:37.158 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-12-15T23:00:37.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:37.259+0000 7f9151c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:37.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:37.267+0000 7f9151c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:37.282 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T23:00:37.282 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T23:00:37.282 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T23:00:37.282 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T23:00:37.282 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T23:00:37.282 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T23:00:37.282 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T23:00:37.282 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T23:00:37.282 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T23:00:37.282 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T23:00:37.282 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T23:00:37.282 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T23:00:37.284 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T23:00:37.285 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:00:37.285 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-12-15T23:00:37.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:37.386+0000 7f106262d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:37.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:37.393+0000 7f106262d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:37.405 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T23:00:37.406 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T23:00:37.406 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T23:00:37.406 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T23:00:37.406 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T23:00:37.406 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T23:00:37.406 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T23:00:37.406 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T23:00:37.406 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T23:00:37.406 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T23:00:37.406 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T23:00:37.406 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T23:00:37.409 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T23:00:37.409 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:00:37.409 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-12-15T23:00:37.464 INFO:tasks.rados.rados.0.smithi071.stdout:1032: done (4 left) 2023-12-15T23:00:37.464 INFO:tasks.rados.rados.0.smithi071.stdout:1033: read oid 2 snap -1 2023-12-15T23:00:37.464 INFO:tasks.rados.rados.0.smithi071.stdout:1033: expect (ObjNum 436 snap 100 seq_num 436) 2023-12-15T23:00:37.464 INFO:tasks.rados.rados.0.smithi071.stdout:1034: read oid 29 snap -1 2023-12-15T23:00:37.464 INFO:tasks.rados.rados.0.smithi071.stdout:1034: expect deleted 2023-12-15T23:00:37.464 INFO:tasks.rados.rados.0.smithi071.stdout:1035: read oid 32 snap 113 2023-12-15T23:00:37.464 INFO:tasks.rados.rados.0.smithi071.stdout:1035: expect (ObjNum 312 snap 81 seq_num 312) 2023-12-15T23:00:37.464 INFO:tasks.rados.rados.0.smithi071.stdout:1036: delete oid 31 current snap is 116 2023-12-15T23:00:37.465 INFO:tasks.rados.rados.0.smithi071.stdout:1030: finishing write tid 1 to smithi07183189-48 2023-12-15T23:00:37.465 INFO:tasks.rados.rados.0.smithi071.stdout:1031: finishing write tid 1 to smithi07183189-30 2023-12-15T23:00:37.465 INFO:tasks.rados.rados.0.smithi071.stdout:1030: finishing write tid 2 to smithi07183189-48 2023-12-15T23:00:37.465 INFO:tasks.rados.rados.0.smithi071.stdout:1031: finishing write tid 2 to smithi07183189-30 2023-12-15T23:00:37.468 INFO:tasks.rados.rados.0.smithi071.stdout:1031: finishing write tid 3 to smithi07183189-30 2023-12-15T23:00:37.468 INFO:tasks.rados.rados.0.smithi071.stdout:1030: finishing write tid 3 to smithi07183189-48 2023-12-15T23:00:37.468 INFO:tasks.rados.rados.0.smithi071.stdout:1030: finishing write tid 4 to smithi07183189-48 2023-12-15T23:00:37.468 INFO:tasks.rados.rados.0.smithi071.stdout:1030: oid 48 updating version 0 to 239 2023-12-15T23:00:37.468 INFO:tasks.rados.rados.0.smithi071.stdout:1030: oid 48 updating version 239 to 240 2023-12-15T23:00:37.468 INFO:tasks.rados.rados.0.smithi071.stdout:1030: oid 48 updating version 240 to 241 2023-12-15T23:00:37.469 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 241 (ObjNum 505 snap 116 seq_num 505) dirty exists 2023-12-15T23:00:37.469 INFO:tasks.rados.rados.0.smithi071.stdout:1030: left oid 48 (ObjNum 505 snap 116 seq_num 505) 2023-12-15T23:00:37.469 INFO:tasks.rados.rados.0.smithi071.stdout:1031: finishing write tid 4 to smithi07183189-30 2023-12-15T23:00:37.469 INFO:tasks.rados.rados.0.smithi071.stdout:1031: oid 30 updating version 0 to 130 2023-12-15T23:00:37.469 INFO:tasks.rados.rados.0.smithi071.stdout:1031: oid 30 version 130 is already newer than 129 2023-12-15T23:00:37.469 INFO:tasks.rados.rados.0.smithi071.stdout:1031: oid 30 updating version 130 to 131 2023-12-15T23:00:37.469 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 131 (ObjNum 506 snap 116 seq_num 506) dirty exists 2023-12-15T23:00:37.469 INFO:tasks.rados.rados.0.smithi071.stdout:1031: left oid 30 (ObjNum 506 snap 116 seq_num 506) 2023-12-15T23:00:37.510 INFO:tasks.rados.rados.0.smithi071.stdout:1033: expect (ObjNum 436 snap 100 seq_num 436) 2023-12-15T23:00:37.538 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:37.533+0000 7fcb55e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:37.542 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:37.542+0000 7fcb55e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:37.557 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T23:00:37.557 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T23:00:37.557 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T23:00:37.557 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T23:00:37.557 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T23:00:37.557 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T23:00:37.557 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T23:00:37.557 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T23:00:37.557 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T23:00:37.557 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T23:00:37.557 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T23:00:37.557 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T23:00:37.559 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T23:00:37.560 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:00:37.560 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-12-15T23:00:37.647 INFO:tasks.rados.rados.0.smithi071.stdout:1028: done (7 left) 2023-12-15T23:00:37.647 INFO:tasks.rados.rados.0.smithi071.stdout:1029: done (6 left) 2023-12-15T23:00:37.647 INFO:tasks.rados.rados.0.smithi071.stdout:1030: done (5 left) 2023-12-15T23:00:37.647 INFO:tasks.rados.rados.0.smithi071.stdout:1031: done (4 left) 2023-12-15T23:00:37.647 INFO:tasks.rados.rados.0.smithi071.stdout:1033: done (3 left) 2023-12-15T23:00:37.647 INFO:tasks.rados.rados.0.smithi071.stdout:1034: done (2 left) 2023-12-15T23:00:37.647 INFO:tasks.rados.rados.0.smithi071.stdout:1036: done (1 left) 2023-12-15T23:00:37.647 INFO:tasks.rados.rados.0.smithi071.stdout:1037: write oid 4 current snap is 116 2023-12-15T23:00:37.647 INFO:tasks.rados.rados.0.smithi071.stdout:1035: expect (ObjNum 312 snap 81 seq_num 312) 2023-12-15T23:00:37.667 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:37.662+0000 7fa39d79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:37.670 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:37.670+0000 7fa39d79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:37.682 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T23:00:37.682 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T23:00:37.683 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T23:00:37.683 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T23:00:37.683 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T23:00:37.683 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T23:00:37.683 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T23:00:37.683 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T23:00:37.683 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T23:00:37.683 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T23:00:37.683 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T23:00:37.683 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T23:00:37.685 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T23:00:37.685 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:00:37.853 INFO:tasks.rados.rados.0.smithi071.stdout:1037: seq_num 508 ranges {537956=705790,1644029=772160} 2023-12-15T23:00:37.859 INFO:tasks.rados.rados.0.smithi071.stdout:1037: writing smithi07183189-4 from 537956 to 1243746 tid 1 2023-12-15T23:00:37.866 INFO:tasks.rados.rados.0.smithi071.stdout:1037: writing smithi07183189-4 from 1644029 to 2416189 tid 2 2023-12-15T23:00:37.867 INFO:tasks.rados.rados.0.smithi071.stdout:1035: done (1 left) 2023-12-15T23:00:37.867 INFO:tasks.rados.rados.0.smithi071.stdout:1038: read oid 7 snap -1 2023-12-15T23:00:37.867 INFO:tasks.rados.rados.0.smithi071.stdout:1038: expect deleted 2023-12-15T23:00:37.867 INFO:tasks.rados.rados.0.smithi071.stdout:1039: snap_remove snap 115 2023-12-15T23:00:38.467 INFO:tasks.rados.rados.0.smithi071.stdout:1039: done (2 left) 2023-12-15T23:00:38.467 INFO:tasks.rados.rados.0.smithi071.stdout:1040: write oid 16 current snap is 116 2023-12-15T23:00:38.467 INFO:tasks.rados.rados.0.smithi071.stdout:1037: finishing write tid 1 to smithi07183189-4 2023-12-15T23:00:38.468 INFO:tasks.rados.rados.0.smithi071.stdout:1040: seq_num 509 ranges {554763=781039,2102511=452252} 2023-12-15T23:00:38.474 INFO:tasks.rados.rados.0.smithi071.stdout:1040: writing smithi07183189-16 from 554763 to 1335802 tid 1 2023-12-15T23:00:38.482 INFO:tasks.rados.rados.0.smithi071.stdout:1040: writing smithi07183189-16 from 2102511 to 2554763 tid 2 2023-12-15T23:00:38.482 INFO:tasks.rados.rados.0.smithi071.stdout:1041: delete oid 42 current snap is 116 2023-12-15T23:00:38.482 INFO:tasks.rados.rados.0.smithi071.stdout:1037: finishing write tid 2 to smithi07183189-4 2023-12-15T23:00:38.482 INFO:tasks.rados.rados.0.smithi071.stdout:1037: finishing write tid 3 to smithi07183189-4 2023-12-15T23:00:38.482 INFO:tasks.rados.rados.0.smithi071.stdout:1037: finishing write tid 4 to smithi07183189-4 2023-12-15T23:00:38.482 INFO:tasks.rados.rados.0.smithi071.stdout:1037: oid 4 updating version 0 to 324 2023-12-15T23:00:38.482 INFO:tasks.rados.rados.0.smithi071.stdout:1037: oid 4 updating version 324 to 326 2023-12-15T23:00:38.482 INFO:tasks.rados.rados.0.smithi071.stdout:1037: oid 4 version 326 is already newer than 325 2023-12-15T23:00:38.482 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 4 v 326 (ObjNum 508 snap 116 seq_num 508) dirty exists 2023-12-15T23:00:38.482 INFO:tasks.rados.rados.0.smithi071.stdout:1037: left oid 4 (ObjNum 508 snap 116 seq_num 508) 2023-12-15T23:00:38.542 INFO:tasks.rados.rados.0.smithi071.stdout:1040: finishing write tid 1 to smithi07183189-16 2023-12-15T23:00:38.569 INFO:tasks.rados.rados.0.smithi071.stdout:1040: finishing write tid 2 to smithi07183189-16 2023-12-15T23:00:38.576 INFO:tasks.rados.rados.0.smithi071.stdout:1037: done (3 left) 2023-12-15T23:00:38.576 INFO:tasks.rados.rados.0.smithi071.stdout:1038: done (2 left) 2023-12-15T23:00:38.577 INFO:tasks.rados.rados.0.smithi071.stdout:1041: done (1 left) 2023-12-15T23:00:38.577 INFO:tasks.rados.rados.0.smithi071.stdout:1042: write oid 18 current snap is 116 2023-12-15T23:00:38.577 INFO:tasks.rados.rados.0.smithi071.stdout:1042: seq_num 511 ranges {588377=647890,1935828=635840} 2023-12-15T23:00:38.583 INFO:tasks.rados.rados.0.smithi071.stdout:1042: writing smithi07183189-18 from 588377 to 1236267 tid 1 2023-12-15T23:00:38.591 INFO:tasks.rados.rados.0.smithi071.stdout:1042: writing smithi07183189-18 from 1935828 to 2571668 tid 2 2023-12-15T23:00:38.591 INFO:tasks.rados.rados.0.smithi071.stdout:1043: write (excl) oid 5 current snap is 116 2023-12-15T23:00:38.591 INFO:tasks.rados.rados.0.smithi071.stdout:1040: finishing write tid 3 to smithi07183189-16 2023-12-15T23:00:38.591 INFO:tasks.rados.rados.0.smithi071.stdout:1043: seq_num 512 ranges {605184=723139,1877957=727227} 2023-12-15T23:00:38.598 INFO:tasks.rados.rados.0.smithi071.stdout:1043: writing smithi07183189-5 from 605184 to 1328323 tid 1 2023-12-15T23:00:38.606 INFO:tasks.rados.rados.0.smithi071.stdout:1043: writing smithi07183189-5 from 1877957 to 2605184 tid 2 2023-12-15T23:00:38.606 INFO:tasks.rados.rados.0.smithi071.stdout:1040: finishing write tid 4 to smithi07183189-16 2023-12-15T23:00:38.606 INFO:tasks.rados.rados.0.smithi071.stdout:1040: oid 16 updating version 0 to 330 2023-12-15T23:00:38.606 INFO:tasks.rados.rados.0.smithi071.stdout:1040: oid 16 updating version 330 to 331 2023-12-15T23:00:38.606 INFO:tasks.rados.rados.0.smithi071.stdout:1040: oid 16 version 331 is already newer than 327 2023-12-15T23:00:38.606 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 16 v 331 (ObjNum 509 snap 116 seq_num 509) dirty exists 2023-12-15T23:00:38.606 INFO:tasks.rados.rados.0.smithi071.stdout:1040: left oid 16 (ObjNum 509 snap 116 seq_num 509) 2023-12-15T23:00:38.606 INFO:tasks.rados.rados.0.smithi071.stdout:1044: read oid 23 snap 113 2023-12-15T23:00:38.606 INFO:tasks.rados.rados.0.smithi071.stdout:1044: expect deleted 2023-12-15T23:00:38.606 INFO:tasks.rados.rados.0.smithi071.stdout:1040: done (3 left) 2023-12-15T23:00:38.606 INFO:tasks.rados.rados.0.smithi071.stdout:1045: delete oid 8 current snap is 116 2023-12-15T23:00:38.620 INFO:tasks.rados.rados.0.smithi071.stdout:1042: finishing write tid 1 to smithi07183189-18 2023-12-15T23:00:38.626 INFO:tasks.rados.rados.0.smithi071.stdout:1043: finishing write tid 1 to smithi07183189-5 2023-12-15T23:00:38.639 INFO:tasks.rados.rados.0.smithi071.stdout:1042: finishing write tid 2 to smithi07183189-18 2023-12-15T23:00:38.651 INFO:tasks.rados.rados.0.smithi071.stdout:1043: finishing write tid 2 to smithi07183189-5 2023-12-15T23:00:38.664 INFO:tasks.rados.rados.0.smithi071.stdout:1043: finishing write tid 3 to smithi07183189-5 2023-12-15T23:00:38.665 INFO:tasks.rados.rados.0.smithi071.stdout:1043: finishing write tid 4 to smithi07183189-5 2023-12-15T23:00:38.666 INFO:tasks.rados.rados.0.smithi071.stdout:1043: oid 5 updating version 0 to 616 2023-12-15T23:00:38.666 INFO:tasks.rados.rados.0.smithi071.stdout:1043: oid 5 version 616 is already newer than 616 2023-12-15T23:00:38.666 INFO:tasks.rados.rados.0.smithi071.stdout:1043: oid 5 version 616 is already newer than 614 2023-12-15T23:00:38.666 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 5 v 616 (ObjNum 512 snap 116 seq_num 512) dirty exists 2023-12-15T23:00:38.666 INFO:tasks.rados.rados.0.smithi071.stdout:1043: left oid 5 (ObjNum 512 snap 116 seq_num 512) 2023-12-15T23:00:38.689 INFO:tasks.rados.rados.0.smithi071.stdout:1042: finishing write tid 3 to smithi07183189-18 2023-12-15T23:00:38.690 INFO:tasks.rados.rados.0.smithi071.stdout:1043: done (3 left) 2023-12-15T23:00:38.690 INFO:tasks.rados.rados.0.smithi071.stdout:1044: done (2 left) 2023-12-15T23:00:38.690 INFO:tasks.rados.rados.0.smithi071.stdout:1045: done (1 left) 2023-12-15T23:00:38.691 INFO:tasks.rados.rados.0.smithi071.stdout:1046: snap_create 2023-12-15T23:00:38.691 INFO:tasks.rados.rados.0.smithi071.stdout:1042: finishing write tid 4 to smithi07183189-18 2023-12-15T23:00:38.691 INFO:tasks.rados.rados.0.smithi071.stdout:1042: oid 18 updating version 0 to 153 2023-12-15T23:00:38.691 INFO:tasks.rados.rados.0.smithi071.stdout:1042: oid 18 version 153 is already newer than 152 2023-12-15T23:00:38.691 INFO:tasks.rados.rados.0.smithi071.stdout:1042: oid 18 version 153 is already newer than 151 2023-12-15T23:00:38.691 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 153 (ObjNum 511 snap 116 seq_num 511) dirty exists 2023-12-15T23:00:38.691 INFO:tasks.rados.rados.0.smithi071.stdout:1042: left oid 18 (ObjNum 511 snap 116 seq_num 511) 2023-12-15T23:00:38.909 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T23:00:39.012 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:39.011+0000 7fa9cf39e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:39.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:39.015+0000 7fa9cf39e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:39.471 INFO:tasks.rados.rados.0.smithi071.stdout:1042: done (1 left) 2023-12-15T23:00:39.471 INFO:tasks.rados.rados.0.smithi071.stdout:1046: done (0 left) 2023-12-15T23:00:39.471 INFO:tasks.rados.rados.0.smithi071.stdout:1047: read oid 48 snap -1 2023-12-15T23:00:39.471 INFO:tasks.rados.rados.0.smithi071.stdout:1047: expect (ObjNum 505 snap 116 seq_num 505) 2023-12-15T23:00:39.471 INFO:tasks.rados.rados.0.smithi071.stdout:1048: write (excl) oid 45 current snap is 117 2023-12-15T23:00:39.471 INFO:tasks.rados.rados.0.smithi071.stdout:1048: seq_num 514 ranges {638798=473637,1594921=783167} 2023-12-15T23:00:39.477 INFO:tasks.rados.rados.0.smithi071.stdout:1048: writing smithi07183189-45 from 638798 to 1112435 tid 1 2023-12-15T23:00:39.484 INFO:tasks.rados.rados.0.smithi071.stdout:1048: writing smithi07183189-45 from 1594921 to 2378088 tid 2 2023-12-15T23:00:39.484 INFO:tasks.rados.rados.0.smithi071.stdout:1049: delete oid 9 current snap is 117 2023-12-15T23:00:39.527 INFO:tasks.rados.rados.0.smithi071.stdout:1047: expect (ObjNum 505 snap 116 seq_num 505) 2023-12-15T23:00:39.683 INFO:tasks.rados.rados.0.smithi071.stdout:1047: done (2 left) 2023-12-15T23:00:39.683 INFO:tasks.rados.rados.0.smithi071.stdout:1049: done (1 left) 2023-12-15T23:00:39.683 INFO:tasks.rados.rados.0.smithi071.stdout:1050: read oid 46 snap -1 2023-12-15T23:00:39.683 INFO:tasks.rados.rados.0.smithi071.stdout:1050: expect deleted 2023-12-15T23:00:39.683 INFO:tasks.rados.rados.0.smithi071.stdout:1051: write oid 7 current snap is 117 2023-12-15T23:00:39.683 INFO:tasks.rados.rados.0.smithi071.stdout:1048: finishing write tid 1 to smithi07183189-45 2023-12-15T23:00:39.683 INFO:tasks.rados.rados.0.smithi071.stdout:1048: finishing write tid 2 to smithi07183189-45 2023-12-15T23:00:39.683 INFO:tasks.rados.rados.0.smithi071.stdout:1048: finishing write tid 3 to smithi07183189-45 2023-12-15T23:00:39.684 INFO:tasks.rados.rados.0.smithi071.stdout:1048: finishing write tid 4 to smithi07183189-45 2023-12-15T23:00:39.684 INFO:tasks.rados.rados.0.smithi071.stdout:1048: oid 45 updating version 0 to 618 2023-12-15T23:00:39.684 INFO:tasks.rados.rados.0.smithi071.stdout:1048: oid 45 updating version 618 to 619 2023-12-15T23:00:39.684 INFO:tasks.rados.rados.0.smithi071.stdout:1048: oid 45 updating version 619 to 620 2023-12-15T23:00:39.684 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 45 v 620 (ObjNum 514 snap 117 seq_num 514) dirty exists 2023-12-15T23:00:39.684 INFO:tasks.rados.rados.0.smithi071.stdout:1048: left oid 45 (ObjNum 514 snap 117 seq_num 514) 2023-12-15T23:00:39.684 INFO:tasks.rados.rados.0.smithi071.stdout:1051: seq_num 516 ranges {672412=624135,1995532=676880} 2023-12-15T23:00:39.690 INFO:tasks.rados.rados.0.smithi071.stdout:1051: writing smithi07183189-7 from 672412 to 1296547 tid 1 2023-12-15T23:00:39.698 INFO:tasks.rados.rados.0.smithi071.stdout:1051: writing smithi07183189-7 from 1995532 to 2672412 tid 2 2023-12-15T23:00:39.698 INFO:tasks.rados.rados.0.smithi071.stdout:1048: done (2 left) 2023-12-15T23:00:39.698 INFO:tasks.rados.rados.0.smithi071.stdout:1052: read oid 6 snap 116 2023-12-15T23:00:39.698 INFO:tasks.rados.rados.0.smithi071.stdout:1052: expect (ObjNum 375 snap 91 seq_num 375) 2023-12-15T23:00:39.698 INFO:tasks.rados.rados.0.smithi071.stdout:1053: delete oid 10 current snap is 117 2023-12-15T23:00:39.718 INFO:tasks.rados.rados.0.smithi071.stdout:1050: done (3 left) 2023-12-15T23:00:39.718 INFO:tasks.rados.rados.0.smithi071.stdout:1053: done (2 left) 2023-12-15T23:00:39.718 INFO:tasks.rados.rados.0.smithi071.stdout:1054: snap_create 2023-12-15T23:00:39.719 INFO:tasks.rados.rados.0.smithi071.stdout:1052: expect (ObjNum 375 snap 91 seq_num 375) 2023-12-15T23:00:39.928 INFO:tasks.rados.rados.0.smithi071.stdout:1051: finishing write tid 1 to smithi07183189-7 2023-12-15T23:00:39.928 INFO:tasks.rados.rados.0.smithi071.stdout:1051: finishing write tid 2 to smithi07183189-7 2023-12-15T23:00:39.928 INFO:tasks.rados.rados.0.smithi071.stdout:1051: finishing write tid 3 to smithi07183189-7 2023-12-15T23:00:39.928 INFO:tasks.rados.rados.0.smithi071.stdout:1051: finishing write tid 4 to smithi07183189-7 2023-12-15T23:00:39.928 INFO:tasks.rados.rados.0.smithi071.stdout:1051: oid 7 updating version 0 to 332 2023-12-15T23:00:39.928 INFO:tasks.rados.rados.0.smithi071.stdout:1051: oid 7 updating version 332 to 334 2023-12-15T23:00:39.928 INFO:tasks.rados.rados.0.smithi071.stdout:1051: oid 7 version 334 is already newer than 333 2023-12-15T23:00:39.928 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 334 (ObjNum 516 snap 117 seq_num 516) dirty exists 2023-12-15T23:00:39.928 INFO:tasks.rados.rados.0.smithi071.stdout:1051: left oid 7 (ObjNum 516 snap 117 seq_num 516) 2023-12-15T23:00:40.474 INFO:tasks.rados.rados.0.smithi071.stdout:1051: done (2 left) 2023-12-15T23:00:40.474 INFO:tasks.rados.rados.0.smithi071.stdout:1052: done (1 left) 2023-12-15T23:00:40.474 INFO:tasks.rados.rados.0.smithi071.stdout:1054: done (0 left) 2023-12-15T23:00:40.474 INFO:tasks.rados.rados.0.smithi071.stdout:1055: write (excl) oid 38 current snap is 118 2023-12-15T23:00:40.475 INFO:tasks.rados.rados.0.smithi071.stdout:1055: seq_num 518 ranges {706026=490986,1828849=790505} 2023-12-15T23:00:40.480 INFO:tasks.rados.rados.0.smithi071.stdout:1055: writing smithi07183189-38 from 706026 to 1197012 tid 1 2023-12-15T23:00:40.489 INFO:tasks.rados.rados.0.smithi071.stdout:1055: writing smithi07183189-38 from 1828849 to 2619354 tid 2 2023-12-15T23:00:40.489 INFO:tasks.rados.rados.0.smithi071.stdout:1056: read oid 25 snap -1 2023-12-15T23:00:40.489 INFO:tasks.rados.rados.0.smithi071.stdout:1056: expect (ObjNum 502 snap 114 seq_num 502) 2023-12-15T23:00:40.489 INFO:tasks.rados.rados.0.smithi071.stdout:1057: read oid 17 snap -1 2023-12-15T23:00:40.489 INFO:tasks.rados.rados.0.smithi071.stdout:1057: expect deleted 2023-12-15T23:00:40.489 INFO:tasks.rados.rados.0.smithi071.stdout:1058: read oid 43 snap -1 2023-12-15T23:00:40.489 INFO:tasks.rados.rados.0.smithi071.stdout:1058: expect (ObjNum 494 snap 113 seq_num 494) 2023-12-15T23:00:40.490 INFO:tasks.rados.rados.0.smithi071.stdout:1059: read oid 18 snap 116 2023-12-15T23:00:40.490 INFO:tasks.rados.rados.0.smithi071.stdout:1059: expect (ObjNum 511 snap 116 seq_num 511) 2023-12-15T23:00:40.490 INFO:tasks.rados.rados.0.smithi071.stdout:1060: snap_create 2023-12-15T23:00:40.504 INFO:tasks.rados.rados.0.smithi071.stdout:1058: expect (ObjNum 494 snap 113 seq_num 494) 2023-12-15T23:00:40.664 INFO:tasks.rados.rados.0.smithi071.stdout:1059: expect (ObjNum 511 snap 116 seq_num 511) 2023-12-15T23:00:40.805 INFO:tasks.rados.rados.0.smithi071.stdout:1056: expect (ObjNum 502 snap 114 seq_num 502) 2023-12-15T23:00:40.962 INFO:tasks.rados.rados.0.smithi071.stdout:1055: finishing write tid 1 to smithi07183189-38 2023-12-15T23:00:40.962 INFO:tasks.rados.rados.0.smithi071.stdout:1055: finishing write tid 2 to smithi07183189-38 2023-12-15T23:00:40.963 INFO:tasks.rados.rados.0.smithi071.stdout:1055: finishing write tid 3 to smithi07183189-38 2023-12-15T23:00:40.963 INFO:tasks.rados.rados.0.smithi071.stdout:1055: finishing write tid 4 to smithi07183189-38 2023-12-15T23:00:40.963 INFO:tasks.rados.rados.0.smithi071.stdout:1055: oid 38 updating version 0 to 158 2023-12-15T23:00:40.963 INFO:tasks.rados.rados.0.smithi071.stdout:1055: oid 38 version 158 is already newer than 157 2023-12-15T23:00:40.963 INFO:tasks.rados.rados.0.smithi071.stdout:1055: oid 38 updating version 158 to 159 2023-12-15T23:00:40.963 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 38 v 159 (ObjNum 518 snap 118 seq_num 518) dirty exists 2023-12-15T23:00:40.963 INFO:tasks.rados.rados.0.smithi071.stdout:1055: left oid 38 (ObjNum 518 snap 118 seq_num 518) 2023-12-15T23:00:41.084 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T23:00:41.190 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:41.188+0000 7f5e5d4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:41.193 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:41.193+0000 7f5e5d4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:41.477 INFO:tasks.rados.rados.0.smithi071.stdout:1055: done (5 left) 2023-12-15T23:00:41.477 INFO:tasks.rados.rados.0.smithi071.stdout:1056: done (4 left) 2023-12-15T23:00:41.478 INFO:tasks.rados.rados.0.smithi071.stdout:1057: done (3 left) 2023-12-15T23:00:41.478 INFO:tasks.rados.rados.0.smithi071.stdout:1058: done (2 left) 2023-12-15T23:00:41.478 INFO:tasks.rados.rados.0.smithi071.stdout:1059: done (1 left) 2023-12-15T23:00:41.478 INFO:tasks.rados.rados.0.smithi071.stdout:1060: done (0 left) 2023-12-15T23:00:41.478 INFO:tasks.rados.rados.0.smithi071.stdout:1061: snap_create 2023-12-15T23:00:42.279 INFO:tasks.rados.rados.0.smithi071.stdout:1061: done (0 left) 2023-12-15T23:00:42.279 INFO:tasks.rados.rados.0.smithi071.stdout:1062: read oid 24 snap 116 2023-12-15T23:00:42.279 INFO:tasks.rados.rados.0.smithi071.stdout:1062: expect (ObjNum 472 snap 110 seq_num 472) 2023-12-15T23:00:42.280 INFO:tasks.rados.rados.0.smithi071.stdout:1063: snap_create 2023-12-15T23:00:42.337 INFO:tasks.rados.rados.0.smithi071.stdout:1062: expect (ObjNum 472 snap 110 seq_num 472) 2023-12-15T23:00:42.686 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T23:00:42.686 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T23:00:42.687 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T23:00:42.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:42.788+0000 7fc7e715e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:42.794 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:42.795+0000 7fc7e715e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:42.969 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T23:00:42.969 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-12-15T23:00:42.981 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":374,"stamp":"2023-12-15T23:00:42.559472+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":74442669,"num_objects":342,"num_object_clones":248,"num_object_copies":1026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":65,"num_read":2022,"num_read_kb":1425984,"num_write":1719,"num_write_kb":464432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3433,"ondisk_log_size":3433,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":318332,"kb_used_data":177756,"kb_used_omap":0,"kb_used_meta":140544,"kb_avail":377169028,"statfs":{"total":386547056640,"available":386221084672,"internally_reserved":0,"allocated":182022144,"data_stored":187474766,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":143917056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8184844,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":42,"num_read_kb":32381,"num_write":30,"num_write_kb":8647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000854"},"pg_stats":[{"pgid":"3.f","version":"246'98","reported_seq":596,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-12-15T23:00:40.510000+0000","last_change":"2023-12-15T23:00:38.496096+0000","last_active":"2023-12-15T23:00:40.510000+0000","last_peered":"2023-12-15T23:00:40.510000+0000","last_clean":"2023-12-15T23:00:40.510000+0000","last_became_active":"2023-12-15T22:56:39.503797+0000","last_became_peered":"2023-12-15T22:56:39.503797+0000","last_unstale":"2023-12-15T23:00:40.510000+0000","last_undegraded":"2023-12-15T23:00:40.510000+0000","last_fullsized":"2023-12-15T23:00:40.510000+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3243784,"num_objects":12,"num_object_clones":8,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":97,"num_read_kb":83323,"num_write":40,"num_write_kb":11941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":477,"reported_epoch":251,"state":"active+clean+snaptrim","last_fresh":"2023-12-15T23:00:39.575069+0000","last_change":"2023-12-15T23:00:39.575069+0000","last_active":"2023-12-15T23:00:39.575069+0000","last_peered":"2023-12-15T23:00:39.575069+0000","last_clean":"2023-12-15T23:00:39.575069+0000","last_became_active":"2023-12-15T22:56:39.552856+0000","last_became_peered":"2023-12-15T22:56:39.552856+0000","last_unstale":"2023-12-15T23:00:39.575069+0000","last_undegraded":"2023-12-15T23:00:39.575069+0000","last_fullsized":"2023-12-15T23:00:39.575069+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"250'331","reported_seq":771,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-12-15T23:00:39.502531+0000","last_change":"2023-12-15T23:00:38.581639+0000","last_active":"2023-12-15T23:00:39.502531+0000","last_peered":"2023-12-15T23:00:39.502531+0000","last_clean":"2023-12-15T23:00:39.502531+0000","last_became_active":"2023-12-15T22:56:39.495490+0000","last_became_peered":"2023-12-15T22:56:39.495490+0000","last_unstale":"2023-12-15T23:00:39.502531+0000","last_undegraded":"2023-12-15T23:00:39.502531+0000","last_fullsized":"2023-12-15T23:00:39.502531+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7529833,"num_objects":35,"num_object_clones":25,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":255,"num_read_kb":218217,"num_write":181,"num_write_kb":52317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"251'315","reported_seq":728,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-12-15T23:00:40.510665+0000","last_change":"2023-12-15T23:00:38.498151+0000","last_active":"2023-12-15T23:00:40.510665+0000","last_peered":"2023-12-15T23:00:40.510665+0000","last_clean":"2023-12-15T23:00:40.510665+0000","last_became_active":"2023-12-15T22:56:39.579234+0000","last_became_peered":"2023-12-15T22:56:39.579234+0000","last_unstale":"2023-12-15T23:00:40.510665+0000","last_undegraded":"2023-12-15T23:00:40.510665+0000","last_fullsized":"2023-12-15T23:00:40.510665+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":31,"num_object_clones":28,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":159,"num_read_kb":112881,"num_write":157,"num_write_kb":45977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"245'175","reported_seq":611,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-12-15T23:00:40.487087+0000","last_change":"2023-12-15T23:00:38.645297+0000","last_active":"2023-12-15T23:00:40.487087+0000","last_peered":"2023-12-15T23:00:40.487087+0000","last_clean":"2023-12-15T23:00:40.487087+0000","last_became_active":"2023-12-15T22:56:39.572176+0000","last_became_peered":"2023-12-15T22:56:39.572176+0000","last_unstale":"2023-12-15T23:00:40.487087+0000","last_undegraded":"2023-12-15T23:00:40.487087+0000","last_fullsized":"2023-12-15T23:00:40.487087+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":12,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":75,"num_read_kb":45675,"num_write":76,"num_write_kb":22702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"250'153","reported_seq":606,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-12-15T23:00:40.528580+0000","last_change":"2023-12-15T23:00:38.491711+0000","last_active":"2023-12-15T23:00:40.528580+0000","last_peered":"2023-12-15T23:00:40.528580+0000","last_clean":"2023-12-15T23:00:40.528580+0000","last_became_active":"2023-12-15T22:56:39.516361+0000","last_became_peered":"2023-12-15T22:56:39.516361+0000","last_unstale":"2023-12-15T23:00:40.528580+0000","last_undegraded":"2023-12-15T23:00:40.528580+0000","last_fullsized":"2023-12-15T23:00:40.528580+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5916229,"num_objects":15,"num_object_clones":10,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":72,"num_read_kb":56862,"num_write":65,"num_write_kb":19150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"251'620","reported_seq":882,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-12-15T23:00:40.507514+0000","last_change":"2023-12-15T23:00:38.492319+0000","last_active":"2023-12-15T23:00:40.507514+0000","last_peered":"2023-12-15T23:00:40.507514+0000","last_clean":"2023-12-15T23:00:40.507514+0000","last_became_active":"2023-12-15T22:56:39.519748+0000","last_became_peered":"2023-12-15T22:56:39.519748+0000","last_unstale":"2023-12-15T23:00:40.507514+0000","last_undegraded":"2023-12-15T23:00:40.507514+0000","last_fullsized":"2023-12-15T23:00:40.507514+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":620,"log_dups_size":0,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16891563,"num_objects":63,"num_object_clones":48,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":261,"num_read_kb":159828,"num_write":270,"num_write_kb":77412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"245'155","reported_seq":626,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-12-15T23:00:39.501952+0000","last_change":"2023-12-15T23:00:38.497876+0000","last_active":"2023-12-15T23:00:39.501952+0000","last_peered":"2023-12-15T23:00:39.501952+0000","last_clean":"2023-12-15T23:00:39.501952+0000","last_became_active":"2023-12-15T22:56:39.543179+0000","last_became_peered":"2023-12-15T22:56:39.543179+0000","last_unstale":"2023-12-15T23:00:39.501952+0000","last_undegraded":"2023-12-15T23:00:39.501952+0000","last_fullsized":"2023-12-15T23:00:39.501952+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6773485,"num_objects":17,"num_object_clones":11,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":135,"num_read_kb":115076,"num_write":87,"num_write_kb":23778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":520,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-12-15T23:00:40.483455+0000","last_change":"2023-12-15T22:56:32.607017+0000","last_active":"2023-12-15T23:00:40.483455+0000","last_peered":"2023-12-15T23:00:40.483455+0000","last_clean":"2023-12-15T23:00:40.483455+0000","last_became_active":"2023-12-15T22:56:32.602384+0000","last_became_peered":"2023-12-15T22:56:32.602384+0000","last_unstale":"2023-12-15T23:00:40.483455+0000","last_undegraded":"2023-12-15T23:00:40.483455+0000","last_fullsized":"2023-12-15T23:00:40.483455+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_clean_scrub_stamp":"2023-12-15T22:56:31.525572+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":375,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-12-15T23:00:39.506074+0000","last_change":"2023-12-15T22:56:33.627481+0000","last_active":"2023-12-15T23:00:39.506074+0000","last_peered":"2023-12-15T23:00:39.506074+0000","last_clean":"2023-12-15T23:00:39.506074+0000","last_became_active":"2023-12-15T22:56:33.624689+0000","last_became_peered":"2023-12-15T22:56:33.624689+0000","last_unstale":"2023-12-15T23:00:39.506074+0000","last_undegraded":"2023-12-15T23:00:39.506074+0000","last_fullsized":"2023-12-15T23:00:39.506074+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"250'247","reported_seq":720,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-12-15T23:00:39.519314+0000","last_change":"2023-12-15T23:00:38.535696+0000","last_active":"2023-12-15T23:00:39.519314+0000","last_peered":"2023-12-15T23:00:39.519314+0000","last_clean":"2023-12-15T23:00:39.519314+0000","last_became_active":"2023-12-15T22:56:39.529359+0000","last_became_peered":"2023-12-15T22:56:39.529359+0000","last_unstale":"2023-12-15T23:00:39.519314+0000","last_undegraded":"2023-12-15T23:00:39.519314+0000","last_fullsized":"2023-12-15T23:00:39.519314+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":247,"log_dups_size":0,"ondisk_log_size":247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756579,"num_objects":30,"num_object_clones":19,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":205,"num_read_kb":148054,"num_write":149,"num_write_kb":43104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":405,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-12-15T23:00:40.506210+0000","last_change":"2023-12-15T22:56:33.655797+0000","last_active":"2023-12-15T23:00:40.506210+0000","last_peered":"2023-12-15T23:00:40.506210+0000","last_clean":"2023-12-15T23:00:40.506210+0000","last_became_active":"2023-12-15T22:56:33.652885+0000","last_became_peered":"2023-12-15T22:56:33.652885+0000","last_unstale":"2023-12-15T23:00:40.506210+0000","last_undegraded":"2023-12-15T23:00:40.506210+0000","last_fullsized":"2023-12-15T23:00:40.506210+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"250'301","reported_seq":703,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-12-15T23:00:40.486418+0000","last_change":"2023-12-15T23:00:38.658010+0000","last_active":"2023-12-15T23:00:40.486418+0000","last_peered":"2023-12-15T23:00:40.486418+0000","last_clean":"2023-12-15T23:00:40.486418+0000","last_became_active":"2023-12-15T22:56:39.543509+0000","last_became_peered":"2023-12-15T22:56:39.543509+0000","last_unstale":"2023-12-15T23:00:40.486418+0000","last_undegraded":"2023-12-15T23:00:40.486418+0000","last_fullsized":"2023-12-15T23:00:40.486418+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":301,"log_dups_size":0,"ondisk_log_size":301,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8437378,"num_objects":33,"num_object_clones":22,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":172,"num_read_kb":126440,"num_write":130,"num_write_kb":40138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":375,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-12-15T23:00:39.501480+0000","last_change":"2023-12-15T22:56:33.600485+0000","last_active":"2023-12-15T23:00:39.501480+0000","last_peered":"2023-12-15T23:00:39.501480+0000","last_clean":"2023-12-15T23:00:39.501480+0000","last_became_active":"2023-12-15T22:56:33.597320+0000","last_became_peered":"2023-12-15T22:56:33.597320+0000","last_unstale":"2023-12-15T23:00:39.501480+0000","last_undegraded":"2023-12-15T23:00:39.501480+0000","last_fullsized":"2023-12-15T23:00:39.501480+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"250'135","reported_seq":594,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-12-15T23:00:40.501228+0000","last_change":"2023-12-15T23:00:38.684914+0000","last_active":"2023-12-15T23:00:40.501228+0000","last_peered":"2023-12-15T23:00:40.501228+0000","last_clean":"2023-12-15T23:00:40.501228+0000","last_became_active":"2023-12-15T22:56:39.560980+0000","last_became_peered":"2023-12-15T22:56:39.560980+0000","last_unstale":"2023-12-15T23:00:40.501228+0000","last_undegraded":"2023-12-15T23:00:40.501228+0000","last_fullsized":"2023-12-15T23:00:40.501228+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4807000,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":78,"num_read_kb":65655,"num_write":62,"num_write_kb":18098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":405,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-12-15T23:00:40.510414+0000","last_change":"2023-12-15T22:56:33.648541+0000","last_active":"2023-12-15T23:00:40.510414+0000","last_peered":"2023-12-15T23:00:40.510414+0000","last_clean":"2023-12-15T23:00:40.510414+0000","last_became_active":"2023-12-15T22:56:33.643681+0000","last_became_peered":"2023-12-15T22:56:33.643681+0000","last_unstale":"2023-12-15T23:00:40.510414+0000","last_undegraded":"2023-12-15T23:00:40.510414+0000","last_fullsized":"2023-12-15T23:00:40.510414+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"245'241","reported_seq":641,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-12-15T23:00:37.524959+0000","last_change":"2023-12-15T23:00:37.527790+0000","last_active":"2023-12-15T23:00:37.524959+0000","last_peered":"2023-12-15T23:00:37.524959+0000","last_clean":"2023-12-15T23:00:37.524959+0000","last_became_active":"2023-12-15T22:56:39.506510+0000","last_became_peered":"2023-12-15T22:56:39.506510+0000","last_unstale":"2023-12-15T23:00:37.524959+0000","last_undegraded":"2023-12-15T23:00:37.524959+0000","last_fullsized":"2023-12-15T23:00:37.524959+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5496054,"num_objects":26,"num_object_clones":18,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":144,"num_read_kb":100919,"num_write":107,"num_write_kb":31471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"242'110","reported_seq":536,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-12-15T23:00:39.493882+0000","last_change":"2023-12-15T23:00:38.514798+0000","last_active":"2023-12-15T23:00:39.493882+0000","last_peered":"2023-12-15T23:00:39.493882+0000","last_clean":"2023-12-15T23:00:39.493882+0000","last_became_active":"2023-12-15T22:56:39.511159+0000","last_became_peered":"2023-12-15T22:56:39.511159+0000","last_unstale":"2023-12-15T23:00:39.493882+0000","last_undegraded":"2023-12-15T23:00:39.493882+0000","last_fullsized":"2023-12-15T23:00:39.493882+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3932904,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":24,"num_read_kb":7342,"num_write":47,"num_write_kb":14375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":365,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-12-15T23:00:37.526157+0000","last_change":"2023-12-15T22:56:33.633890+0000","last_active":"2023-12-15T23:00:37.526157+0000","last_peered":"2023-12-15T23:00:37.526157+0000","last_clean":"2023-12-15T23:00:37.526157+0000","last_became_active":"2023-12-15T22:56:33.631134+0000","last_became_peered":"2023-12-15T22:56:33.631134+0000","last_unstale":"2023-12-15T23:00:37.526157+0000","last_undegraded":"2023-12-15T23:00:37.526157+0000","last_fullsized":"2023-12-15T23:00:37.526157+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"245'173","reported_seq":590,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-12-15T23:00:37.527745+0000","last_change":"2023-12-15T23:00:37.533263+0000","last_active":"2023-12-15T23:00:37.527745+0000","last_peered":"2023-12-15T23:00:37.527745+0000","last_clean":"2023-12-15T23:00:37.527745+0000","last_became_active":"2023-12-15T22:56:39.532730+0000","last_became_peered":"2023-12-15T22:56:39.532730+0000","last_unstale":"2023-12-15T23:00:37.527745+0000","last_undegraded":"2023-12-15T23:00:37.527745+0000","last_fullsized":"2023-12-15T23:00:37.527745+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":14,"num_object_clones":12,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":87,"num_read_kb":65976,"num_write":82,"num_write_kb":25453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":391,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-12-15T23:00:40.486075+0000","last_change":"2023-12-15T22:56:33.655736+0000","last_active":"2023-12-15T23:00:40.486075+0000","last_peered":"2023-12-15T23:00:40.486075+0000","last_clean":"2023-12-15T23:00:40.486075+0000","last_became_active":"2023-12-15T22:56:33.652917+0000","last_became_peered":"2023-12-15T22:56:33.652917+0000","last_unstale":"2023-12-15T23:00:40.486075+0000","last_undegraded":"2023-12-15T23:00:40.486075+0000","last_fullsized":"2023-12-15T23:00:40.486075+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"249'199","reported_seq":649,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-12-15T23:00:40.484748+0000","last_change":"2023-12-15T23:00:38.648004+0000","last_active":"2023-12-15T23:00:40.484748+0000","last_peered":"2023-12-15T23:00:40.484748+0000","last_clean":"2023-12-15T23:00:40.484748+0000","last_became_active":"2023-12-15T22:56:39.557252+0000","last_became_peered":"2023-12-15T22:56:39.557252+0000","last_unstale":"2023-12-15T23:00:40.484748+0000","last_undegraded":"2023-12-15T23:00:40.484748+0000","last_fullsized":"2023-12-15T23:00:40.484748+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":199,"log_dups_size":0,"ondisk_log_size":199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6067492,"num_objects":18,"num_object_clones":12,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":120,"num_read_kb":90251,"num_write":94,"num_write_kb":26111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":365,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-12-15T23:00:37.514314+0000","last_change":"2023-12-15T22:56:33.632348+0000","last_active":"2023-12-15T23:00:37.514314+0000","last_peered":"2023-12-15T23:00:37.514314+0000","last_clean":"2023-12-15T23:00:37.514314+0000","last_became_active":"2023-12-15T22:56:33.629022+0000","last_became_peered":"2023-12-15T22:56:33.629022+0000","last_unstale":"2023-12-15T23:00:37.514314+0000","last_undegraded":"2023-12-15T23:00:37.514314+0000","last_fullsized":"2023-12-15T23:00:37.514314+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"245'83","reported_seq":524,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-12-15T23:00:37.529027+0000","last_change":"2023-12-15T23:00:37.532769+0000","last_active":"2023-12-15T23:00:37.529027+0000","last_peered":"2023-12-15T23:00:37.529027+0000","last_clean":"2023-12-15T23:00:37.529027+0000","last_became_active":"2023-12-15T22:56:39.535145+0000","last_became_peered":"2023-12-15T22:56:39.535145+0000","last_unstale":"2023-12-15T23:00:37.529027+0000","last_undegraded":"2023-12-15T23:00:37.529027+0000","last_fullsized":"2023-12-15T23:00:37.529027+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":6,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":29394,"num_write":35,"num_write_kb":10511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":375,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-12-15T23:00:39.501803+0000","last_change":"2023-12-15T22:56:33.627005+0000","last_active":"2023-12-15T23:00:39.501803+0000","last_peered":"2023-12-15T23:00:39.501803+0000","last_clean":"2023-12-15T23:00:39.501803+0000","last_became_active":"2023-12-15T22:56:33.622181+0000","last_became_peered":"2023-12-15T22:56:33.622181+0000","last_unstale":"2023-12-15T23:00:39.501803+0000","last_undegraded":"2023-12-15T23:00:39.501803+0000","last_fullsized":"2023-12-15T23:00:39.501803+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":73852301,"num_objects":340,"num_object_clones":248,"num_object_copies":1020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":65,"num_read":1919,"num_read_kb":1425893,"num_write":1582,"num_write_kb":462538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3336,"ondisk_log_size":3336,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":79112,"kb_used_data":43968,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94292728,"statfs":{"total":96636764160,"available":96555753472,"internally_reserved":0,"allocated":45023232,"data_stored":46843390,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705913,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":66948,"kb_used_data":31804,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94304892,"statfs":{"total":96636764160,"available":96568209408,"internally_reserved":0,"allocated":32567296,"data_stored":33408461,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705913,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":87760,"kb_used_data":52616,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94284080,"statfs":{"total":96636764160,"available":96546897920,"internally_reserved":0,"allocated":53878784,"data_stored":55196433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":84512,"kb_used_data":49368,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94287328,"statfs":{"total":96636764160,"available":96550223872,"internally_reserved":0,"allocated":50552832,"data_stored":52026482,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-15T23:00:43.261 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T23:00:43.283 INFO:tasks.rados.rados.0.smithi071.stdout:1062: done (1 left) 2023-12-15T23:00:43.284 INFO:tasks.rados.rados.0.smithi071.stdout:1063: done (0 left) 2023-12-15T23:00:43.284 INFO:tasks.rados.rados.0.smithi071.stdout:1064: write (excl) oid 37 current snap is 121 2023-12-15T23:00:43.284 INFO:tasks.rados.rados.0.smithi071.stdout:1064: seq_num 519 ranges {722833=566235,1887331=650516} 2023-12-15T23:00:43.291 INFO:tasks.rados.rados.0.smithi071.stdout:1064: writing smithi07183189-37 from 722833 to 1289068 tid 1 2023-12-15T23:00:43.300 INFO:tasks.rados.rados.0.smithi071.stdout:1064: writing smithi07183189-37 from 1887331 to 2537847 tid 2 2023-12-15T23:00:43.300 INFO:tasks.rados.rados.0.smithi071.stdout:1065: read oid 22 snap -1 2023-12-15T23:00:43.300 INFO:tasks.rados.rados.0.smithi071.stdout:1065: expect (ObjNum 429 snap 100 seq_num 429) 2023-12-15T23:00:43.301 INFO:tasks.rados.rados.0.smithi071.stdout:1066: snap_remove snap 117 2023-12-15T23:00:43.379 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:43.378+0000 7fc09f66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:43.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:43.382+0000 7fc09f66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:44.285 INFO:tasks.rados.rados.0.smithi071.stdout:1066: done (2 left) 2023-12-15T23:00:44.285 INFO:tasks.rados.rados.0.smithi071.stdout:1067: read oid 15 snap -1 2023-12-15T23:00:44.285 INFO:tasks.rados.rados.0.smithi071.stdout:1067: expect deleted 2023-12-15T23:00:44.285 INFO:tasks.rados.rados.0.smithi071.stdout:1068: write (excl) oid 16 current snap is 121 2023-12-15T23:00:44.286 INFO:tasks.rados.rados.0.smithi071.stdout:1064: finishing write tid 1 to smithi07183189-37 2023-12-15T23:00:44.286 INFO:tasks.rados.rados.0.smithi071.stdout:1068: seq_num 520 ranges {739640=641484,1829460=794174} 2023-12-15T23:00:44.292 INFO:tasks.rados.rados.0.smithi071.stdout:1068: writing smithi07183189-16 from 739640 to 1381124 tid 1 2023-12-15T23:00:44.300 INFO:tasks.rados.rados.0.smithi071.stdout:1068: writing smithi07183189-16 from 1829460 to 2623634 tid 2 2023-12-15T23:00:44.300 INFO:tasks.rados.rados.0.smithi071.stdout:1069: read oid 6 snap -1 2023-12-15T23:00:44.300 INFO:tasks.rados.rados.0.smithi071.stdout:1069: expect (ObjNum 375 snap 91 seq_num 375) 2023-12-15T23:00:44.300 INFO:tasks.rados.rados.0.smithi071.stdout:1070: write (excl) oid 30 current snap is 121 2023-12-15T23:00:44.300 INFO:tasks.rados.rados.0.smithi071.stdout:1070: seq_num 521 ranges {756447=716733,1887942=537832} 2023-12-15T23:00:44.308 INFO:tasks.rados.rados.0.smithi071.stdout:1070: writing smithi07183189-30 from 756447 to 1473180 tid 1 2023-12-15T23:00:44.313 INFO:tasks.rados.rados.0.smithi071.stdout:1070: writing smithi07183189-30 from 1887942 to 2425774 tid 2 2023-12-15T23:00:44.314 INFO:tasks.rados.rados.0.smithi071.stdout:1071: write oid 23 current snap is 121 2023-12-15T23:00:44.314 INFO:tasks.rados.rados.0.smithi071.stdout:1065: expect (ObjNum 429 snap 100 seq_num 429) 2023-12-15T23:00:44.483 INFO:tasks.rados.rados.0.smithi071.stdout:1064: finishing write tid 2 to smithi07183189-37 2023-12-15T23:00:44.483 INFO:tasks.rados.rados.0.smithi071.stdout:1064: finishing write tid 3 to smithi07183189-37 2023-12-15T23:00:44.484 INFO:tasks.rados.rados.0.smithi071.stdout:1064: finishing write tid 4 to smithi07183189-37 2023-12-15T23:00:44.484 INFO:tasks.rados.rados.0.smithi071.stdout:1064: oid 37 updating version 0 to 624 2023-12-15T23:00:44.484 INFO:tasks.rados.rados.0.smithi071.stdout:1064: oid 37 version 624 is already newer than 623 2023-12-15T23:00:44.484 INFO:tasks.rados.rados.0.smithi071.stdout:1064: oid 37 version 624 is already newer than 622 2023-12-15T23:00:44.484 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 37 v 624 (ObjNum 519 snap 121 seq_num 519) dirty exists 2023-12-15T23:00:44.484 INFO:tasks.rados.rados.0.smithi071.stdout:1064: left oid 37 (ObjNum 519 snap 121 seq_num 519) 2023-12-15T23:00:44.484 INFO:tasks.rados.rados.0.smithi071.stdout:1071: seq_num 522 ranges {773254=791982,2346424=426830} 2023-12-15T23:00:44.491 INFO:tasks.rados.rados.0.smithi071.stdout:1071: writing smithi07183189-23 from 773254 to 1565236 tid 1 2023-12-15T23:00:44.498 INFO:tasks.rados.rados.0.smithi071.stdout:1071: writing smithi07183189-23 from 2346424 to 2773254 tid 2 2023-12-15T23:00:44.498 INFO:tasks.rados.rados.0.smithi071.stdout:1064: done (6 left) 2023-12-15T23:00:44.499 INFO:tasks.rados.rados.0.smithi071.stdout:1065: done (5 left) 2023-12-15T23:00:44.499 INFO:tasks.rados.rados.0.smithi071.stdout:1072: write oid 18 current snap is 121 2023-12-15T23:00:44.499 INFO:tasks.rados.rados.0.smithi071.stdout:1072: seq_num 523 ranges {790061=467231,2004906=541501} 2023-12-15T23:00:44.505 INFO:tasks.rados.rados.0.smithi071.stdout:1072: writing smithi07183189-18 from 790061 to 1257292 tid 1 2023-12-15T23:00:44.514 INFO:tasks.rados.rados.0.smithi071.stdout:1072: writing smithi07183189-18 from 2004906 to 2546407 tid 2 2023-12-15T23:00:44.514 INFO:tasks.rados.rados.0.smithi071.stdout:1073: write oid 47 current snap is 121 2023-12-15T23:00:44.514 INFO:tasks.rados.rados.0.smithi071.stdout:1073: seq_num 524 ranges {406868=542480,1547035=401512,2690354=116514} 2023-12-15T23:00:44.519 INFO:tasks.rados.rados.0.smithi071.stdout:1073: writing smithi07183189-47 from 406868 to 949348 tid 1 2023-12-15T23:00:44.526 INFO:tasks.rados.rados.0.smithi071.stdout:1073: writing smithi07183189-47 from 1547035 to 1948547 tid 2 2023-12-15T23:00:44.531 INFO:tasks.rados.rados.0.smithi071.stdout:1073: writing smithi07183189-47 from 2690354 to 2806868 tid 3 2023-12-15T23:00:44.531 INFO:tasks.rados.rados.0.smithi071.stdout:1074: snap_remove snap 119 2023-12-15T23:00:45.297 INFO:tasks.rados.rados.0.smithi071.stdout:1074: done (7 left) 2023-12-15T23:00:45.297 INFO:tasks.rados.rados.0.smithi071.stdout:1075: write oid 9 current snap is 121 2023-12-15T23:00:45.297 INFO:tasks.rados.rados.0.smithi071.stdout:1075: seq_num 525 ranges {423675=734082,1721870=545170,2717777=105898} 2023-12-15T23:00:45.303 INFO:tasks.rados.rados.0.smithi071.stdout:1075: writing smithi07183189-9 from 423675 to 1157757 tid 1 2023-12-15T23:00:45.309 INFO:tasks.rados.rados.0.smithi071.stdout:1075: writing smithi07183189-9 from 1721870 to 2267040 tid 2 2023-12-15T23:00:45.313 INFO:tasks.rados.rados.0.smithi071.stdout:1075: writing smithi07183189-9 from 2717777 to 2823675 tid 3 2023-12-15T23:00:45.313 INFO:tasks.rados.rados.0.smithi071.stdout:1076: write (excl) oid 36 current snap is 121 2023-12-15T23:00:45.313 INFO:tasks.rados.rados.0.smithi071.stdout:1069: expect (ObjNum 375 snap 91 seq_num 375) 2023-12-15T23:00:45.451 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T23:00:45.528 INFO:tasks.rados.rados.0.smithi071.stdout:1068: finishing write tid 1 to smithi07183189-16 2023-12-15T23:00:45.528 INFO:tasks.rados.rados.0.smithi071.stdout:1070: finishing write tid 1 to smithi07183189-30 2023-12-15T23:00:45.528 INFO:tasks.rados.rados.0.smithi071.stdout:1070: finishing write tid 2 to smithi07183189-30 2023-12-15T23:00:45.528 INFO:tasks.rados.rados.0.smithi071.stdout:1068: finishing write tid 2 to smithi07183189-16 2023-12-15T23:00:45.528 INFO:tasks.rados.rados.0.smithi071.stdout:1070: finishing write tid 3 to smithi07183189-30 2023-12-15T23:00:45.528 INFO:tasks.rados.rados.0.smithi071.stdout:1070: finishing write tid 4 to smithi07183189-30 2023-12-15T23:00:45.528 INFO:tasks.rados.rados.0.smithi071.stdout:1070: oid 30 updating version 0 to 139 2023-12-15T23:00:45.529 INFO:tasks.rados.rados.0.smithi071.stdout:1070: oid 30 version 139 is already newer than 137 2023-12-15T23:00:45.529 INFO:tasks.rados.rados.0.smithi071.stdout:1070: oid 30 version 139 is already newer than 138 2023-12-15T23:00:45.529 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 139 (ObjNum 521 snap 121 seq_num 521) dirty exists 2023-12-15T23:00:45.529 INFO:tasks.rados.rados.0.smithi071.stdout:1070: left oid 30 (ObjNum 521 snap 121 seq_num 521) 2023-12-15T23:00:45.529 INFO:tasks.rados.rados.0.smithi071.stdout:1068: finishing write tid 3 to smithi07183189-16 2023-12-15T23:00:45.529 INFO:tasks.rados.rados.0.smithi071.stdout:1068: finishing write tid 4 to smithi07183189-16 2023-12-15T23:00:45.529 INFO:tasks.rados.rados.0.smithi071.stdout:1068: oid 16 updating version 0 to 336 2023-12-15T23:00:45.529 INFO:tasks.rados.rados.0.smithi071.stdout:1068: oid 16 updating version 336 to 338 2023-12-15T23:00:45.529 INFO:tasks.rados.rados.0.smithi071.stdout:1068: oid 16 version 338 is already newer than 337 2023-12-15T23:00:45.529 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 16 v 338 (ObjNum 520 snap 121 seq_num 520) dirty exists 2023-12-15T23:00:45.529 INFO:tasks.rados.rados.0.smithi071.stdout:1068: left oid 16 (ObjNum 520 snap 121 seq_num 520) 2023-12-15T23:00:45.529 INFO:tasks.rados.rados.0.smithi071.stdout:1071: finishing write tid 1 to smithi07183189-23 2023-12-15T23:00:45.529 INFO:tasks.rados.rados.0.smithi071.stdout:1073: finishing write tid 1 to smithi07183189-47 2023-12-15T23:00:45.529 INFO:tasks.rados.rados.0.smithi071.stdout:1076: seq_num 526 ranges {440482=409331,1380352=405181,2461553=378929} 2023-12-15T23:00:45.534 INFO:tasks.rados.rados.0.smithi071.stdout:1076: writing smithi07183189-36 from 440482 to 849813 tid 1 2023-12-15T23:00:45.540 INFO:tasks.rados.rados.0.smithi071.stdout:1076: writing smithi07183189-36 from 1380352 to 1785533 tid 2 2023-12-15T23:00:45.549 INFO:tasks.rados.rados.0.smithi071.stdout:1076: writing smithi07183189-36 from 2461553 to 2840482 tid 3 2023-12-15T23:00:45.549 INFO:tasks.rados.rados.0.smithi071.stdout:1067: done (8 left) 2023-12-15T23:00:45.549 INFO:tasks.rados.rados.0.smithi071.stdout:1068: done (7 left) 2023-12-15T23:00:45.549 INFO:tasks.rados.rados.0.smithi071.stdout:1069: done (6 left) 2023-12-15T23:00:45.549 INFO:tasks.rados.rados.0.smithi071.stdout:1070: done (5 left) 2023-12-15T23:00:45.549 INFO:tasks.rados.rados.0.smithi071.stdout:1077: read oid 20 snap -1 2023-12-15T23:00:45.549 INFO:tasks.rados.rados.0.smithi071.stdout:1077: expect (ObjNum 480 snap 112 seq_num 480) 2023-12-15T23:00:45.549 INFO:tasks.rados.rados.0.smithi071.stdout:1078: read oid 37 snap -1 2023-12-15T23:00:45.549 INFO:tasks.rados.rados.0.smithi071.stdout:1078: expect (ObjNum 519 snap 121 seq_num 519) 2023-12-15T23:00:45.549 INFO:tasks.rados.rados.0.smithi071.stdout:1079: delete oid 5 current snap is 121 2023-12-15T23:00:45.550 INFO:tasks.rados.rados.0.smithi071.stdout:1072: finishing write tid 1 to smithi07183189-18 2023-12-15T23:00:45.550 INFO:tasks.rados.rados.0.smithi071.stdout:1071: finishing write tid 2 to smithi07183189-23 2023-12-15T23:00:45.550 INFO:tasks.rados.rados.0.smithi071.stdout:1073: finishing write tid 2 to smithi07183189-47 2023-12-15T23:00:45.550 INFO:tasks.rados.rados.0.smithi071.stdout:1071: finishing write tid 3 to smithi07183189-23 2023-12-15T23:00:45.550 INFO:tasks.rados.rados.0.smithi071.stdout:1071: finishing write tid 4 to smithi07183189-23 2023-12-15T23:00:45.550 INFO:tasks.rados.rados.0.smithi071.stdout:1071: oid 23 updating version 0 to 201 2023-12-15T23:00:45.550 INFO:tasks.rados.rados.0.smithi071.stdout:1071: oid 23 updating version 201 to 203 2023-12-15T23:00:45.550 INFO:tasks.rados.rados.0.smithi071.stdout:1071: oid 23 version 203 is already newer than 202 2023-12-15T23:00:45.550 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 23 v 203 (ObjNum 522 snap 121 seq_num 522) dirty exists 2023-12-15T23:00:45.550 INFO:tasks.rados.rados.0.smithi071.stdout:1071: left oid 23 (ObjNum 522 snap 121 seq_num 522) 2023-12-15T23:00:45.550 INFO:tasks.rados.rados.0.smithi071.stdout:1072: finishing write tid 2 to smithi07183189-18 2023-12-15T23:00:45.550 INFO:tasks.rados.rados.0.smithi071.stdout:1072: finishing write tid 3 to smithi07183189-18 2023-12-15T23:00:45.550 INFO:tasks.rados.rados.0.smithi071.stdout:1072: finishing write tid 4 to smithi07183189-18 2023-12-15T23:00:45.550 INFO:tasks.rados.rados.0.smithi071.stdout:1072: oid 18 updating version 0 to 155 2023-12-15T23:00:45.551 INFO:tasks.rados.rados.0.smithi071.stdout:1072: oid 18 updating version 155 to 157 2023-12-15T23:00:45.551 INFO:tasks.rados.rados.0.smithi071.stdout:1072: oid 18 version 157 is already newer than 157 2023-12-15T23:00:45.551 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 18 v 157 (ObjNum 523 snap 121 seq_num 523) dirty exists 2023-12-15T23:00:45.551 INFO:tasks.rados.rados.0.smithi071.stdout:1072: left oid 18 (ObjNum 523 snap 121 seq_num 523) 2023-12-15T23:00:45.551 INFO:tasks.rados.rados.0.smithi071.stdout:1073: finishing write tid 3 to smithi07183189-47 2023-12-15T23:00:45.551 INFO:tasks.rados.rados.0.smithi071.stdout:1073: finishing write tid 4 to smithi07183189-47 2023-12-15T23:00:45.551 INFO:tasks.rados.rados.0.smithi071.stdout:1073: finishing write tid 5 to smithi07183189-47 2023-12-15T23:00:45.551 INFO:tasks.rados.rados.0.smithi071.stdout:1073: oid 47 updating version 0 to 176 2023-12-15T23:00:45.551 INFO:tasks.rados.rados.0.smithi071.stdout:1073: oid 47 updating version 176 to 179 2023-12-15T23:00:45.551 INFO:tasks.rados.rados.0.smithi071.stdout:1073: oid 47 version 179 is already newer than 177 2023-12-15T23:00:45.551 INFO:tasks.rados.rados.0.smithi071.stdout:1073: oid 47 version 179 is already newer than 179 2023-12-15T23:00:45.551 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 179 (ObjNum 524 snap 121 seq_num 524) dirty exists 2023-12-15T23:00:45.551 INFO:tasks.rados.rados.0.smithi071.stdout:1073: left oid 47 (ObjNum 524 snap 121 seq_num 524) 2023-12-15T23:00:45.551 INFO:tasks.rados.rados.0.smithi071.stdout:1075: finishing write tid 1 to smithi07183189-9 2023-12-15T23:00:45.551 INFO:tasks.rados.rados.0.smithi071.stdout:1075: finishing write tid 2 to smithi07183189-9 2023-12-15T23:00:45.551 INFO:tasks.rados.rados.0.smithi071.stdout:1075: finishing write tid 3 to smithi07183189-9 2023-12-15T23:00:45.552 INFO:tasks.rados.rados.0.smithi071.stdout:1075: finishing write tid 4 to smithi07183189-9 2023-12-15T23:00:45.552 INFO:tasks.rados.rados.0.smithi071.stdout:1075: finishing write tid 5 to smithi07183189-9 2023-12-15T23:00:45.552 INFO:tasks.rados.rados.0.smithi071.stdout:1075: oid 9 updating version 0 to 319 2023-12-15T23:00:45.552 INFO:tasks.rados.rados.0.smithi071.stdout:1075: oid 9 version 319 is already newer than 319 2023-12-15T23:00:45.552 INFO:tasks.rados.rados.0.smithi071.stdout:1075: oid 9 version 319 is already newer than 316 2023-12-15T23:00:45.552 INFO:tasks.rados.rados.0.smithi071.stdout:1075: oid 9 version 319 is already newer than 317 2023-12-15T23:00:45.552 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 319 (ObjNum 525 snap 121 seq_num 525) dirty exists 2023-12-15T23:00:45.552 INFO:tasks.rados.rados.0.smithi071.stdout:1075: left oid 9 (ObjNum 525 snap 121 seq_num 525) 2023-12-15T23:00:45.564 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:45.563+0000 7f481d51e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:45.567 INFO:tasks.rados.rados.0.smithi071.stdout:1077: expect (ObjNum 480 snap 112 seq_num 480) 2023-12-15T23:00:45.568 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:45.568+0000 7f481d51e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:45.691 INFO:tasks.rados.rados.0.smithi071.stdout:1071: done (7 left) 2023-12-15T23:00:45.691 INFO:tasks.rados.rados.0.smithi071.stdout:1072: done (6 left) 2023-12-15T23:00:45.691 INFO:tasks.rados.rados.0.smithi071.stdout:1073: done (5 left) 2023-12-15T23:00:45.691 INFO:tasks.rados.rados.0.smithi071.stdout:1075: done (4 left) 2023-12-15T23:00:45.691 INFO:tasks.rados.rados.0.smithi071.stdout:1077: done (3 left) 2023-12-15T23:00:45.691 INFO:tasks.rados.rados.0.smithi071.stdout:1079: done (2 left) 2023-12-15T23:00:45.691 INFO:tasks.rados.rados.0.smithi071.stdout:1080: snap_create 2023-12-15T23:00:45.691 INFO:tasks.rados.rados.0.smithi071.stdout:1078: expect (ObjNum 519 snap 121 seq_num 519) 2023-12-15T23:00:45.849 INFO:tasks.rados.rados.0.smithi071.stdout:1076: finishing write tid 1 to smithi07183189-36 2023-12-15T23:00:45.849 INFO:tasks.rados.rados.0.smithi071.stdout:1076: finishing write tid 2 to smithi07183189-36 2023-12-15T23:00:45.849 INFO:tasks.rados.rados.0.smithi071.stdout:1076: finishing write tid 3 to smithi07183189-36 2023-12-15T23:00:45.849 INFO:tasks.rados.rados.0.smithi071.stdout:1076: finishing write tid 4 to smithi07183189-36 2023-12-15T23:00:45.849 INFO:tasks.rados.rados.0.smithi071.stdout:1076: finishing write tid 5 to smithi07183189-36 2023-12-15T23:00:45.849 INFO:tasks.rados.rados.0.smithi071.stdout:1076: oid 36 updating version 0 to 343 2023-12-15T23:00:45.849 INFO:tasks.rados.rados.0.smithi071.stdout:1076: oid 36 version 343 is already newer than 339 2023-12-15T23:00:45.850 INFO:tasks.rados.rados.0.smithi071.stdout:1076: oid 36 version 343 is already newer than 340 2023-12-15T23:00:45.850 INFO:tasks.rados.rados.0.smithi071.stdout:1076: oid 36 updating version 343 to 344 2023-12-15T23:00:45.850 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 344 (ObjNum 526 snap 121 seq_num 526) dirty exists 2023-12-15T23:00:45.850 INFO:tasks.rados.rados.0.smithi071.stdout:1076: left oid 36 (ObjNum 526 snap 121 seq_num 526) 2023-12-15T23:00:46.295 INFO:tasks.rados.rados.0.smithi071.stdout:1076: done (2 left) 2023-12-15T23:00:46.296 INFO:tasks.rados.rados.0.smithi071.stdout:1078: done (1 left) 2023-12-15T23:00:46.296 INFO:tasks.rados.rados.0.smithi071.stdout:1080: done (0 left) 2023-12-15T23:00:46.296 INFO:tasks.rados.rados.0.smithi071.stdout:1081: delete oid 17 current snap is 122 2023-12-15T23:00:46.316 INFO:tasks.rados.rados.0.smithi071.stdout:1081: done (0 left) 2023-12-15T23:00:46.316 INFO:tasks.rados.rados.0.smithi071.stdout:1082: write oid 1 current snap is 122 2023-12-15T23:00:46.316 INFO:tasks.rados.rados.0.smithi071.stdout:1082: seq_num 529 ranges {490903=635078,1839445=552508} 2023-12-15T23:00:46.322 INFO:tasks.rados.rados.0.smithi071.stdout:1082: writing smithi07183189-1 from 490903 to 1125981 tid 1 2023-12-15T23:00:46.330 INFO:tasks.rados.rados.0.smithi071.stdout:1082: writing smithi07183189-1 from 1839445 to 2391953 tid 2 2023-12-15T23:00:46.330 INFO:tasks.rados.rados.0.smithi071.stdout:1083: snap_remove snap 121 2023-12-15T23:00:47.301 INFO:tasks.rados.rados.0.smithi071.stdout:1083: done (1 left) 2023-12-15T23:00:47.301 INFO:tasks.rados.rados.0.smithi071.stdout:1084: read oid 7 snap 113 2023-12-15T23:00:47.301 INFO:tasks.rados.rados.0.smithi071.stdout:1084: expect deleted 2023-12-15T23:00:47.301 INFO:tasks.rados.rados.0.smithi071.stdout:1085: write (excl) oid 9 current snap is 122 2023-12-15T23:00:47.301 INFO:tasks.rados.rados.0.smithi071.stdout:1082: finishing write tid 1 to smithi07183189-1 2023-12-15T23:00:47.301 INFO:tasks.rados.rados.0.smithi071.stdout:1082: finishing write tid 2 to smithi07183189-1 2023-12-15T23:00:47.302 INFO:tasks.rados.rados.0.smithi071.stdout:1082: finishing write tid 3 to smithi07183189-1 2023-12-15T23:00:47.302 INFO:tasks.rados.rados.0.smithi071.stdout:1082: finishing write tid 4 to smithi07183189-1 2023-12-15T23:00:47.302 INFO:tasks.rados.rados.0.smithi071.stdout:1082: oid 1 updating version 0 to 100 2023-12-15T23:00:47.302 INFO:tasks.rados.rados.0.smithi071.stdout:1082: oid 1 updating version 100 to 101 2023-12-15T23:00:47.302 INFO:tasks.rados.rados.0.smithi071.stdout:1082: oid 1 version 101 is already newer than 100 2023-12-15T23:00:47.302 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 1 v 101 (ObjNum 529 snap 122 seq_num 529) dirty exists 2023-12-15T23:00:47.302 INFO:tasks.rados.rados.0.smithi071.stdout:1082: left oid 1 (ObjNum 529 snap 122 seq_num 529) 2023-12-15T23:00:47.302 INFO:tasks.rados.rados.0.smithi071.stdout:1085: seq_num 530 ranges {507710=710327,1897927=412519,2854892=52818} 2023-12-15T23:00:47.309 INFO:tasks.rados.rados.0.smithi071.stdout:1085: writing smithi07183189-9 from 507710 to 1218037 tid 1 2023-12-15T23:00:47.316 INFO:tasks.rados.rados.0.smithi071.stdout:1085: writing smithi07183189-9 from 1897927 to 2310446 tid 2 2023-12-15T23:00:47.321 INFO:tasks.rados.rados.0.smithi071.stdout:1085: writing smithi07183189-9 from 2854892 to 2907710 tid 3 2023-12-15T23:00:47.321 INFO:tasks.rados.rados.0.smithi071.stdout:1082: done (2 left) 2023-12-15T23:00:47.321 INFO:tasks.rados.rados.0.smithi071.stdout:1086: read oid 13 snap -1 2023-12-15T23:00:47.321 INFO:tasks.rados.rados.0.smithi071.stdout:1086: expect (ObjNum 469 snap 110 seq_num 469) 2023-12-15T23:00:47.321 INFO:tasks.rados.rados.0.smithi071.stdout:1087: read oid 12 snap -1 2023-12-15T23:00:47.321 INFO:tasks.rados.rados.0.smithi071.stdout:1087: expect deleted 2023-12-15T23:00:47.321 INFO:tasks.rados.rados.0.smithi071.stdout:1088: read oid 24 snap -1 2023-12-15T23:00:47.321 INFO:tasks.rados.rados.0.smithi071.stdout:1088: expect (ObjNum 472 snap 110 seq_num 472) 2023-12-15T23:00:47.321 INFO:tasks.rados.rados.0.smithi071.stdout:1089: read oid 45 snap -1 2023-12-15T23:00:47.322 INFO:tasks.rados.rados.0.smithi071.stdout:1089: expect (ObjNum 514 snap 117 seq_num 514) 2023-12-15T23:00:47.322 INFO:tasks.rados.rados.0.smithi071.stdout:1090: write oid 30 current snap is 122 2023-12-15T23:00:47.322 INFO:tasks.rados.rados.0.smithi071.stdout:1090: seq_num 531 ranges {524517=785576,1956409=556177} 2023-12-15T23:00:47.330 INFO:tasks.rados.rados.0.smithi071.stdout:1090: writing smithi07183189-30 from 524517 to 1310093 tid 1 2023-12-15T23:00:47.338 INFO:tasks.rados.rados.0.smithi071.stdout:1090: writing smithi07183189-30 from 1956409 to 2512586 tid 2 2023-12-15T23:00:47.339 INFO:tasks.rados.rados.0.smithi071.stdout:1091: read oid 32 snap -1 2023-12-15T23:00:47.339 INFO:tasks.rados.rados.0.smithi071.stdout:1091: expect (ObjNum 312 snap 81 seq_num 312) 2023-12-15T23:00:47.339 INFO:tasks.rados.rados.0.smithi071.stdout:1092: read oid 26 snap -1 2023-12-15T23:00:47.339 INFO:tasks.rados.rados.0.smithi071.stdout:1092: expect (ObjNum 452 snap 104 seq_num 452) 2023-12-15T23:00:47.339 INFO:tasks.rados.rados.0.smithi071.stdout:1093: snap_remove snap 118 2023-12-15T23:00:47.717 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T23:00:47.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:47.820+0000 7f0ca759e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:47.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:47.825+0000 7f0ca759e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:47.985 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T23:00:47.986 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2023-12-15T23:00:48.094 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:48.088+0000 7f3ad6e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:48.097 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:48.097+0000 7f3ad6e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:48.301 INFO:teuthology.orchestra.run.smithi071.stderr:reweighted osd.0 to 1 (10000) 2023-12-15T23:00:48.305 INFO:tasks.rados.rados.0.smithi071.stdout:1093: done (9 left) 2023-12-15T23:00:48.305 INFO:tasks.rados.rados.0.smithi071.stdout:1094: read oid 21 snap -1 2023-12-15T23:00:48.305 INFO:tasks.rados.rados.0.smithi071.stdout:1094: expect deleted 2023-12-15T23:00:48.305 INFO:tasks.rados.rados.0.smithi071.stdout:1088: expect (ObjNum 472 snap 110 seq_num 472) 2023-12-15T23:00:48.318 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2023-12-15T23:00:48.428 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:48.423+0000 7fbd2371e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:48.431 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:48.431+0000 7fbd2371e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:48.525 INFO:tasks.rados.rados.0.smithi071.stdout:1086: expect (ObjNum 469 snap 110 seq_num 469) 2023-12-15T23:00:48.681 INFO:tasks.rados.rados.0.smithi071.stdout:1091: expect (ObjNum 312 snap 81 seq_num 312) 2023-12-15T23:00:48.886 INFO:tasks.rados.rados.0.smithi071.stdout:1089: expect (ObjNum 514 snap 117 seq_num 514) 2023-12-15T23:00:49.041 INFO:tasks.rados.rados.0.smithi071.stdout:1090: finishing write tid 1 to smithi07183189-30 2023-12-15T23:00:49.041 INFO:tasks.rados.rados.0.smithi071.stdout:1085: finishing write tid 1 to smithi07183189-9 2023-12-15T23:00:49.041 INFO:tasks.rados.rados.0.smithi071.stdout:1095: read oid 38 snap -1 2023-12-15T23:00:49.041 INFO:tasks.rados.rados.0.smithi071.stdout:1095: expect (ObjNum 518 snap 118 seq_num 518) 2023-12-15T23:00:49.042 INFO:tasks.rados.rados.0.smithi071.stdout:1084: done (10 left) 2023-12-15T23:00:49.042 INFO:tasks.rados.rados.0.smithi071.stdout:1086: done (9 left) 2023-12-15T23:00:49.042 INFO:tasks.rados.rados.0.smithi071.stdout:1087: done (8 left) 2023-12-15T23:00:49.042 INFO:tasks.rados.rados.0.smithi071.stdout:1088: done (7 left) 2023-12-15T23:00:49.042 INFO:tasks.rados.rados.0.smithi071.stdout:1089: done (6 left) 2023-12-15T23:00:49.042 INFO:tasks.rados.rados.0.smithi071.stdout:1091: done (5 left) 2023-12-15T23:00:49.042 INFO:tasks.rados.rados.0.smithi071.stdout:1096: write oid 47 current snap is 122 2023-12-15T23:00:49.042 INFO:tasks.rados.rados.0.smithi071.stdout:1092: expect (ObjNum 452 snap 104 seq_num 452) 2023-12-15T23:00:49.186 INFO:tasks.rados.rados.0.smithi071.stdout:1090: finishing write tid 2 to smithi07183189-30 2023-12-15T23:00:49.186 INFO:tasks.rados.rados.0.smithi071.stdout:1085: finishing write tid 2 to smithi07183189-9 2023-12-15T23:00:49.186 INFO:tasks.rados.rados.0.smithi071.stdout:1090: finishing write tid 3 to smithi07183189-30 2023-12-15T23:00:49.186 INFO:tasks.rados.rados.0.smithi071.stdout:1096: seq_num 532 ranges {541324=460825,1498538=416188,2393385=547939} 2023-12-15T23:00:49.191 INFO:tasks.rados.rados.0.smithi071.stdout:1096: writing smithi07183189-47 from 541324 to 1002149 tid 1 2023-12-15T23:00:49.196 INFO:tasks.rados.rados.0.smithi071.stdout:1096: writing smithi07183189-47 from 1498538 to 1914726 tid 2 2023-12-15T23:00:49.204 INFO:tasks.rados.rados.0.smithi071.stdout:1096: writing smithi07183189-47 from 2393385 to 2941324 tid 3 2023-12-15T23:00:49.204 INFO:tasks.rados.rados.0.smithi071.stdout:1092: done (5 left) 2023-12-15T23:00:49.204 INFO:tasks.rados.rados.0.smithi071.stdout:1097: write oid 7 current snap is 122 2023-12-15T23:00:49.204 INFO:tasks.rados.rados.0.smithi071.stdout:1085: finishing write tid 3 to smithi07183189-9 2023-12-15T23:00:49.204 INFO:tasks.rados.rados.0.smithi071.stdout:1090: finishing write tid 4 to smithi07183189-30 2023-12-15T23:00:49.204 INFO:tasks.rados.rados.0.smithi071.stdout:1090: oid 30 updating version 0 to 143 2023-12-15T23:00:49.204 INFO:tasks.rados.rados.0.smithi071.stdout:1090: oid 30 version 143 is already newer than 142 2023-12-15T23:00:49.204 INFO:tasks.rados.rados.0.smithi071.stdout:1090: oid 30 updating version 143 to 144 2023-12-15T23:00:49.205 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 30 v 144 (ObjNum 531 snap 122 seq_num 531) dirty exists 2023-12-15T23:00:49.205 INFO:tasks.rados.rados.0.smithi071.stdout:1090: left oid 30 (ObjNum 531 snap 122 seq_num 531) 2023-12-15T23:00:49.205 INFO:tasks.rados.rados.0.smithi071.stdout:1085: finishing write tid 4 to smithi07183189-9 2023-12-15T23:00:49.205 INFO:tasks.rados.rados.0.smithi071.stdout:1085: finishing write tid 5 to smithi07183189-9 2023-12-15T23:00:49.205 INFO:tasks.rados.rados.0.smithi071.stdout:1085: oid 9 updating version 0 to 322 2023-12-15T23:00:49.205 INFO:tasks.rados.rados.0.smithi071.stdout:1085: oid 9 updating version 322 to 323 2023-12-15T23:00:49.205 INFO:tasks.rados.rados.0.smithi071.stdout:1085: oid 9 version 323 is already newer than 320 2023-12-15T23:00:49.205 INFO:tasks.rados.rados.0.smithi071.stdout:1085: oid 9 version 323 is already newer than 322 2023-12-15T23:00:49.205 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 323 (ObjNum 530 snap 122 seq_num 530) dirty exists 2023-12-15T23:00:49.205 INFO:tasks.rados.rados.0.smithi071.stdout:1085: left oid 9 (ObjNum 530 snap 122 seq_num 530) 2023-12-15T23:00:49.205 INFO:tasks.rados.rados.0.smithi071.stdout:1095: expect (ObjNum 518 snap 118 seq_num 518) 2023-12-15T23:00:49.302 INFO:teuthology.orchestra.run.smithi071.stderr:reweighted osd.1 to 1 (10000) 2023-12-15T23:00:49.316 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2023-12-15T23:00:49.355 INFO:tasks.rados.rados.0.smithi071.stdout:1096: finishing write tid 1 to smithi07183189-47 2023-12-15T23:00:49.355 INFO:tasks.rados.rados.0.smithi071.stdout:1096: finishing write tid 2 to smithi07183189-47 2023-12-15T23:00:49.355 INFO:tasks.rados.rados.0.smithi071.stdout:1096: finishing write tid 3 to smithi07183189-47 2023-12-15T23:00:49.355 INFO:tasks.rados.rados.0.smithi071.stdout:1096: finishing write tid 4 to smithi07183189-47 2023-12-15T23:00:49.355 INFO:tasks.rados.rados.0.smithi071.stdout:1096: finishing write tid 5 to smithi07183189-47 2023-12-15T23:00:49.355 INFO:tasks.rados.rados.0.smithi071.stdout:1096: oid 47 updating version 0 to 180 2023-12-15T23:00:49.356 INFO:tasks.rados.rados.0.smithi071.stdout:1096: oid 47 updating version 180 to 183 2023-12-15T23:00:49.356 INFO:tasks.rados.rados.0.smithi071.stdout:1096: oid 47 version 183 is already newer than 181 2023-12-15T23:00:49.356 INFO:tasks.rados.rados.0.smithi071.stdout:1096: oid 47 version 183 is already newer than 182 2023-12-15T23:00:49.356 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 47 v 183 (ObjNum 532 snap 122 seq_num 532) dirty exists 2023-12-15T23:00:49.356 INFO:tasks.rados.rados.0.smithi071.stdout:1096: left oid 47 (ObjNum 532 snap 122 seq_num 532) 2023-12-15T23:00:49.356 INFO:tasks.rados.rados.0.smithi071.stdout:1097: seq_num 533 ranges {558131=652427,1673373=559846,2820808=137323} 2023-12-15T23:00:49.362 INFO:tasks.rados.rados.0.smithi071.stdout:1097: writing smithi07183189-7 from 558131 to 1210558 tid 1 2023-12-15T23:00:49.367 INFO:tasks.rados.rados.0.smithi071.stdout:1097: writing smithi07183189-7 from 1673373 to 2233219 tid 2 2023-12-15T23:00:49.372 INFO:tasks.rados.rados.0.smithi071.stdout:1097: writing smithi07183189-7 from 2820808 to 2958131 tid 3 2023-12-15T23:00:49.372 INFO:tasks.rados.rados.0.smithi071.stdout:1085: done (5 left) 2023-12-15T23:00:49.372 INFO:tasks.rados.rados.0.smithi071.stdout:1090: done (4 left) 2023-12-15T23:00:49.372 INFO:tasks.rados.rados.0.smithi071.stdout:1094: done (3 left) 2023-12-15T23:00:49.372 INFO:tasks.rados.rados.0.smithi071.stdout:1095: done (2 left) 2023-12-15T23:00:49.372 INFO:tasks.rados.rados.0.smithi071.stdout:1096: done (1 left) 2023-12-15T23:00:49.372 INFO:tasks.rados.rados.0.smithi071.stdout:1098: write (excl) oid 13 current snap is 122 2023-12-15T23:00:49.372 INFO:tasks.rados.rados.0.smithi071.stdout:1098: seq_num 534 ranges {574938=727676,1731855=703504,2848231=126707} 2023-12-15T23:00:49.379 INFO:tasks.rados.rados.0.smithi071.stdout:1098: writing smithi07183189-13 from 574938 to 1302614 tid 1 2023-12-15T23:00:49.385 INFO:tasks.rados.rados.0.smithi071.stdout:1098: writing smithi07183189-13 from 1731855 to 2435359 tid 2 2023-12-15T23:00:49.388 INFO:tasks.rados.rados.0.smithi071.stdout:1098: writing smithi07183189-13 from 2848231 to 2974938 tid 3 2023-12-15T23:00:49.388 INFO:tasks.rados.rados.0.smithi071.stdout:1099: snap_create 2023-12-15T23:00:49.408 INFO:tasks.rados.rados.0.smithi071.stdout:1097: finishing write tid 1 to smithi07183189-7 2023-12-15T23:00:49.455 INFO:tasks.rados.rados.0.smithi071.stdout:1097: finishing write tid 2 to smithi07183189-7 2023-12-15T23:00:49.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:49.452+0000 7fefbbf9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:49.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:49.461+0000 7fefbbf9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:49.466 INFO:tasks.rados.rados.0.smithi071.stdout:1098: finishing write tid 1 to smithi07183189-13 2023-12-15T23:00:49.468 INFO:tasks.rados.rados.0.smithi071.stdout:1097: finishing write tid 3 to smithi07183189-7 2023-12-15T23:00:49.493 INFO:tasks.rados.rados.0.smithi071.stdout:1098: finishing write tid 2 to smithi07183189-13 2023-12-15T23:00:49.496 INFO:tasks.rados.rados.0.smithi071.stdout:1098: finishing write tid 3 to smithi07183189-13 2023-12-15T23:00:49.497 INFO:tasks.rados.rados.0.smithi071.stdout:1097: finishing write tid 4 to smithi07183189-7 2023-12-15T23:00:49.499 INFO:tasks.rados.rados.0.smithi071.stdout:1097: finishing write tid 5 to smithi07183189-7 2023-12-15T23:00:49.499 INFO:tasks.rados.rados.0.smithi071.stdout:1097: oid 7 updating version 0 to 350 2023-12-15T23:00:49.499 INFO:tasks.rados.rados.0.smithi071.stdout:1097: oid 7 updating version 350 to 351 2023-12-15T23:00:49.499 INFO:tasks.rados.rados.0.smithi071.stdout:1097: oid 7 updating version 351 to 352 2023-12-15T23:00:49.499 INFO:tasks.rados.rados.0.smithi071.stdout:1097: oid 7 version 352 is already newer than 349 2023-12-15T23:00:49.499 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 352 (ObjNum 533 snap 122 seq_num 533) dirty exists 2023-12-15T23:00:49.499 INFO:tasks.rados.rados.0.smithi071.stdout:1097: left oid 7 (ObjNum 533 snap 122 seq_num 533) 2023-12-15T23:00:49.509 INFO:tasks.rados.rados.0.smithi071.stdout:1098: finishing write tid 4 to smithi07183189-13 2023-12-15T23:00:49.510 INFO:tasks.rados.rados.0.smithi071.stdout:1098: finishing write tid 5 to smithi07183189-13 2023-12-15T23:00:49.511 INFO:tasks.rados.rados.0.smithi071.stdout:1098: oid 13 updating version 0 to 303 2023-12-15T23:00:49.511 INFO:tasks.rados.rados.0.smithi071.stdout:1098: oid 13 updating version 303 to 305 2023-12-15T23:00:49.511 INFO:tasks.rados.rados.0.smithi071.stdout:1098: oid 13 version 305 is already newer than 305 2023-12-15T23:00:49.511 INFO:tasks.rados.rados.0.smithi071.stdout:1098: oid 13 updating version 305 to 306 2023-12-15T23:00:49.511 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 306 (ObjNum 534 snap 122 seq_num 534) dirty exists 2023-12-15T23:00:49.511 INFO:tasks.rados.rados.0.smithi071.stdout:1098: left oid 13 (ObjNum 534 snap 122 seq_num 534) 2023-12-15T23:00:49.895 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T23:00:49.998 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:49.996+0000 7f876591e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:50.001 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:50.001+0000 7f876591e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:50.311 INFO:teuthology.orchestra.run.smithi071.stderr:reweighted osd.2 to 1 (10000) 2023-12-15T23:00:50.312 INFO:tasks.rados.rados.0.smithi071.stdout:1097: done (2 left) 2023-12-15T23:00:50.313 INFO:tasks.rados.rados.0.smithi071.stdout:1098: done (1 left) 2023-12-15T23:00:50.313 INFO:tasks.rados.rados.0.smithi071.stdout:1099: done (0 left) 2023-12-15T23:00:50.313 INFO:tasks.rados.rados.0.smithi071.stdout:1100: write oid 2 current snap is 123 2023-12-15T23:00:50.313 INFO:tasks.rados.rados.0.smithi071.stdout:1100: seq_num 535 ranges {591745=402925,1790337=563515} 2023-12-15T23:00:50.318 INFO:tasks.rados.rados.0.smithi071.stdout:1100: writing smithi07183189-2 from 591745 to 994670 tid 1 2023-12-15T23:00:50.325 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2023-12-15T23:00:50.326 INFO:tasks.rados.rados.0.smithi071.stdout:1100: writing smithi07183189-2 from 1790337 to 2353852 tid 2 2023-12-15T23:00:50.326 INFO:tasks.rados.rados.0.smithi071.stdout:1101: read oid 6 snap -1 2023-12-15T23:00:50.326 INFO:tasks.rados.rados.0.smithi071.stdout:1101: expect (ObjNum 375 snap 91 seq_num 375) 2023-12-15T23:00:50.326 INFO:tasks.rados.rados.0.smithi071.stdout:1102: delete oid 13 current snap is 123 2023-12-15T23:00:50.362 INFO:tasks.rados.rados.0.smithi071.stdout:1101: expect (ObjNum 375 snap 91 seq_num 375) 2023-12-15T23:00:50.451 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:50.446+0000 7f67a5f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:50.454 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:50.454+0000 7f67a5f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:50.593 INFO:tasks.rados.rados.0.smithi071.stdout:1100: finishing write tid 1 to smithi07183189-2 2023-12-15T23:00:50.593 INFO:tasks.rados.rados.0.smithi071.stdout:1100: finishing write tid 2 to smithi07183189-2 2023-12-15T23:00:50.593 INFO:tasks.rados.rados.0.smithi071.stdout:1100: finishing write tid 3 to smithi07183189-2 2023-12-15T23:00:50.593 INFO:tasks.rados.rados.0.smithi071.stdout:1100: finishing write tid 4 to smithi07183189-2 2023-12-15T23:00:50.593 INFO:tasks.rados.rados.0.smithi071.stdout:1100: oid 2 updating version 0 to 165 2023-12-15T23:00:50.593 INFO:tasks.rados.rados.0.smithi071.stdout:1100: oid 2 version 165 is already newer than 164 2023-12-15T23:00:50.594 INFO:tasks.rados.rados.0.smithi071.stdout:1100: oid 2 version 165 is already newer than 163 2023-12-15T23:00:50.594 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 2 v 165 (ObjNum 535 snap 123 seq_num 535) dirty exists 2023-12-15T23:00:50.594 INFO:tasks.rados.rados.0.smithi071.stdout:1100: left oid 2 (ObjNum 535 snap 123 seq_num 535) 2023-12-15T23:00:50.594 INFO:tasks.rados.rados.0.smithi071.stdout:1100: done (2 left) 2023-12-15T23:00:50.594 INFO:tasks.rados.rados.0.smithi071.stdout:1101: done (1 left) 2023-12-15T23:00:50.594 INFO:tasks.rados.rados.0.smithi071.stdout:1102: done (0 left) 2023-12-15T23:00:50.594 INFO:tasks.rados.rados.0.smithi071.stdout:1103: snap_remove snap 122 2023-12-15T23:00:51.314 INFO:teuthology.orchestra.run.smithi071.stderr:reweighted osd.3 to 1 (10000) 2023-12-15T23:00:51.320 INFO:tasks.rados.rados.0.smithi071.stdout:1103: done (0 left) 2023-12-15T23:00:51.320 INFO:tasks.rados.rados.0.smithi071.stdout:1104: snap_remove snap 116 2023-12-15T23:00:51.328 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2023-12-15T23:00:51.328 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T23:00:51.473 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:51.470+0000 7fbc730ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:51.476 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:51.476+0000 7fbc730ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:51.650 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T23:00:51.650 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-12-15T23:00:51.662 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":387,"stamp":"2023-12-15T23:00:51.315666+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":84023022,"num_objects":356,"num_object_clones":259,"num_object_copies":1068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":65,"num_read":2125,"num_read_kb":1518105,"num_write":1794,"num_write_kb":485313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3530,"ondisk_log_size":3530,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":372656,"kb_used_data":227332,"kb_used_omap":0,"kb_used_meta":140544,"kb_avail":377114704,"statfs":{"total":386547056640,"available":386165456896,"internally_reserved":0,"allocated":232787968,"data_stored":247714763,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":143917056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2185075,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":43758,"num_write":37,"num_write_kb":9971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.017137"},"pg_stats":[{"pgid":"3.f","version":"258'101","reported_seq":627,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-12-15T23:00:50.367191+0000","last_change":"2023-12-15T23:00:48.658797+0000","last_active":"2023-12-15T23:00:50.367191+0000","last_peered":"2023-12-15T23:00:50.367191+0000","last_clean":"2023-12-15T23:00:50.367191+0000","last_became_active":"2023-12-15T22:56:39.503797+0000","last_became_peered":"2023-12-15T22:56:39.503797+0000","last_unstale":"2023-12-15T23:00:50.367191+0000","last_undegraded":"2023-12-15T23:00:50.367191+0000","last_fullsized":"2023-12-15T23:00:50.367191+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6134687,"num_objects":12,"num_object_clones":8,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":104,"num_read_kb":90404,"num_write":44,"num_write_kb":13102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":496,"reported_epoch":260,"state":"active+clean","last_fresh":"2023-12-15T23:00:48.650884+0000","last_change":"2023-12-15T23:00:48.658556+0000","last_active":"2023-12-15T23:00:48.650884+0000","last_peered":"2023-12-15T23:00:48.650884+0000","last_clean":"2023-12-15T23:00:48.650884+0000","last_became_active":"2023-12-15T22:56:39.552856+0000","last_became_peered":"2023-12-15T22:56:39.552856+0000","last_unstale":"2023-12-15T23:00:48.650884+0000","last_undegraded":"2023-12-15T23:00:48.650884+0000","last_fullsized":"2023-12-15T23:00:48.650884+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"261'352","reported_seq":814,"reported_epoch":261,"state":"active+clean","last_fresh":"2023-12-15T23:00:49.469116+0000","last_change":"2023-12-15T23:00:48.344788+0000","last_active":"2023-12-15T23:00:49.469116+0000","last_peered":"2023-12-15T23:00:49.469116+0000","last_clean":"2023-12-15T23:00:49.469116+0000","last_became_active":"2023-12-15T22:56:39.495490+0000","last_became_peered":"2023-12-15T22:56:39.495490+0000","last_unstale":"2023-12-15T23:00:49.469116+0000","last_undegraded":"2023-12-15T23:00:49.469116+0000","last_fullsized":"2023-12-15T23:00:49.469116+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13513323,"num_objects":39,"num_object_clones":27,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":267,"num_read_kb":224831,"num_write":199,"num_write_kb":57479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"259'323","reported_seq":762,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-12-15T23:00:50.370390+0000","last_change":"2023-12-15T23:00:48.656617+0000","last_active":"2023-12-15T23:00:50.370390+0000","last_peered":"2023-12-15T23:00:50.370390+0000","last_clean":"2023-12-15T23:00:50.370390+0000","last_became_active":"2023-12-15T22:56:39.579234+0000","last_became_peered":"2023-12-15T22:56:39.579234+0000","last_unstale":"2023-12-15T23:00:50.370390+0000","last_undegraded":"2023-12-15T23:00:50.370390+0000","last_fullsized":"2023-12-15T23:00:50.370390+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2907710,"num_objects":31,"num_object_clones":28,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":162,"num_read_kb":112883,"num_write":167,"num_write_kb":48480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"260'183","reported_seq":641,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-12-15T23:00:50.347600+0000","last_change":"2023-12-15T23:00:48.322909+0000","last_active":"2023-12-15T23:00:50.347600+0000","last_peered":"2023-12-15T23:00:50.347600+0000","last_clean":"2023-12-15T23:00:50.347600+0000","last_became_active":"2023-12-15T22:56:39.572176+0000","last_became_peered":"2023-12-15T22:56:39.572176+0000","last_unstale":"2023-12-15T23:00:50.347600+0000","last_undegraded":"2023-12-15T23:00:50.347600+0000","last_fullsized":"2023-12-15T23:00:50.347600+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2941324,"num_objects":18,"num_object_clones":12,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":77,"num_read_kb":45677,"num_write":86,"num_write_kb":25133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"262'165","reported_seq":638,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-12-15T23:00:50.426673+0000","last_change":"2023-12-15T23:00:48.676042+0000","last_active":"2023-12-15T23:00:50.426673+0000","last_peered":"2023-12-15T23:00:50.426673+0000","last_clean":"2023-12-15T23:00:50.426673+0000","last_became_active":"2023-12-15T22:56:39.516361+0000","last_became_peered":"2023-12-15T22:56:39.516361+0000","last_unstale":"2023-12-15T23:00:50.426673+0000","last_undegraded":"2023-12-15T23:00:50.426673+0000","last_fullsized":"2023-12-15T23:00:50.426673+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5781806,"num_objects":17,"num_object_clones":12,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":74,"num_read_kb":56864,"num_write":73,"num_write_kb":21081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"260'634","reported_seq":920,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-12-15T23:00:50.338883+0000","last_change":"2023-12-15T23:00:48.678049+0000","last_active":"2023-12-15T23:00:50.338883+0000","last_peered":"2023-12-15T23:00:50.338883+0000","last_clean":"2023-12-15T23:00:50.338883+0000","last_became_active":"2023-12-15T22:56:39.519748+0000","last_became_peered":"2023-12-15T22:56:39.519748+0000","last_unstale":"2023-12-15T23:00:50.338883+0000","last_undegraded":"2023-12-15T23:00:50.338883+0000","last_fullsized":"2023-12-15T23:00:50.338883+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":634,"log_dups_size":0,"ondisk_log_size":634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14555291,"num_objects":65,"num_object_clones":50,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":282,"num_read_kb":177634,"num_write":274,"num_write_kb":78601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"256'161","reported_seq":661,"reported_epoch":261,"state":"active+clean","last_fresh":"2023-12-15T23:00:49.318640+0000","last_change":"2023-12-15T23:00:48.321538+0000","last_active":"2023-12-15T23:00:49.318640+0000","last_peered":"2023-12-15T23:00:49.318640+0000","last_clean":"2023-12-15T23:00:49.318640+0000","last_became_active":"2023-12-15T22:56:39.543179+0000","last_became_peered":"2023-12-15T22:56:39.543179+0000","last_unstale":"2023-12-15T23:00:49.318640+0000","last_undegraded":"2023-12-15T23:00:49.318640+0000","last_fullsized":"2023-12-15T23:00:49.318640+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7260888,"num_objects":18,"num_object_clones":12,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":155,"num_read_kb":134340,"num_write":91,"num_write_kb":25030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":536,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-12-15T23:00:50.349006+0000","last_change":"2023-12-15T22:56:32.607017+0000","last_active":"2023-12-15T23:00:50.349006+0000","last_peered":"2023-12-15T23:00:50.349006+0000","last_clean":"2023-12-15T23:00:50.349006+0000","last_became_active":"2023-12-15T22:56:32.602384+0000","last_became_peered":"2023-12-15T22:56:32.602384+0000","last_unstale":"2023-12-15T23:00:50.349006+0000","last_undegraded":"2023-12-15T23:00:50.349006+0000","last_fullsized":"2023-12-15T23:00:50.349006+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_clean_scrub_stamp":"2023-12-15T22:56:31.525572+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":393,"reported_epoch":261,"state":"active+clean","last_fresh":"2023-12-15T23:00:49.320577+0000","last_change":"2023-12-15T22:56:33.627481+0000","last_active":"2023-12-15T23:00:49.320577+0000","last_peered":"2023-12-15T23:00:49.320577+0000","last_clean":"2023-12-15T23:00:49.320577+0000","last_became_active":"2023-12-15T22:56:33.624689+0000","last_became_peered":"2023-12-15T22:56:33.624689+0000","last_unstale":"2023-12-15T23:00:49.320577+0000","last_undegraded":"2023-12-15T23:00:49.320577+0000","last_fullsized":"2023-12-15T23:00:49.320577+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"251'248","reported_seq":742,"reported_epoch":261,"state":"active+clean","last_fresh":"2023-12-15T23:00:49.317617+0000","last_change":"2023-12-15T23:00:48.318145+0000","last_active":"2023-12-15T23:00:49.317617+0000","last_peered":"2023-12-15T23:00:49.317617+0000","last_clean":"2023-12-15T23:00:49.317617+0000","last_became_active":"2023-12-15T22:56:39.529359+0000","last_became_peered":"2023-12-15T22:56:39.529359+0000","last_unstale":"2023-12-15T23:00:49.317617+0000","last_undegraded":"2023-12-15T23:00:49.317617+0000","last_fullsized":"2023-12-15T23:00:49.317617+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756579,"num_objects":30,"num_object_clones":19,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":205,"num_read_kb":148054,"num_write":149,"num_write_kb":43104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":425,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-12-15T23:00:50.356183+0000","last_change":"2023-12-15T22:56:33.655797+0000","last_active":"2023-12-15T23:00:50.356183+0000","last_peered":"2023-12-15T23:00:50.356183+0000","last_clean":"2023-12-15T23:00:50.356183+0000","last_became_active":"2023-12-15T22:56:33.652885+0000","last_became_peered":"2023-12-15T22:56:33.652885+0000","last_unstale":"2023-12-15T23:00:50.356183+0000","last_undegraded":"2023-12-15T23:00:50.356183+0000","last_fullsized":"2023-12-15T23:00:50.356183+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"260'306","reported_seq":732,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-12-15T23:00:50.340483+0000","last_change":"2023-12-15T23:00:48.327524+0000","last_active":"2023-12-15T23:00:50.340483+0000","last_peered":"2023-12-15T23:00:50.340483+0000","last_clean":"2023-12-15T23:00:50.340483+0000","last_became_active":"2023-12-15T22:56:39.543509+0000","last_became_peered":"2023-12-15T22:56:39.543509+0000","last_unstale":"2023-12-15T23:00:50.340483+0000","last_undegraded":"2023-12-15T23:00:50.340483+0000","last_fullsized":"2023-12-15T23:00:50.340483+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":306,"log_dups_size":0,"ondisk_log_size":306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4554895,"num_objects":35,"num_object_clones":24,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":180,"num_read_kb":134405,"num_write":135,"num_write_kb":41661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":393,"reported_epoch":261,"state":"active+clean","last_fresh":"2023-12-15T23:00:49.318036+0000","last_change":"2023-12-15T22:56:33.600485+0000","last_active":"2023-12-15T23:00:49.318036+0000","last_peered":"2023-12-15T23:00:49.318036+0000","last_clean":"2023-12-15T23:00:49.318036+0000","last_became_active":"2023-12-15T22:56:33.597320+0000","last_became_peered":"2023-12-15T22:56:33.597320+0000","last_unstale":"2023-12-15T23:00:49.318036+0000","last_undegraded":"2023-12-15T23:00:49.318036+0000","last_fullsized":"2023-12-15T23:00:49.318036+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"260'146","reported_seq":622,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-12-15T23:00:50.347753+0000","last_change":"2023-12-15T23:00:48.343368+0000","last_active":"2023-12-15T23:00:50.347753+0000","last_peered":"2023-12-15T23:00:50.347753+0000","last_clean":"2023-12-15T23:00:50.347753+0000","last_became_active":"2023-12-15T22:56:39.560980+0000","last_became_peered":"2023-12-15T22:56:39.560980+0000","last_unstale":"2023-12-15T23:00:50.347753+0000","last_undegraded":"2023-12-15T23:00:50.347753+0000","last_fullsized":"2023-12-15T23:00:50.347753+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5227175,"num_objects":13,"num_object_clones":10,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":81,"num_read_kb":65657,"num_write":70,"num_write_kb":20636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":425,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-12-15T23:00:50.369939+0000","last_change":"2023-12-15T22:56:33.648541+0000","last_active":"2023-12-15T23:00:50.369939+0000","last_peered":"2023-12-15T23:00:50.369939+0000","last_clean":"2023-12-15T23:00:50.369939+0000","last_became_active":"2023-12-15T22:56:33.643681+0000","last_became_peered":"2023-12-15T22:56:33.643681+0000","last_unstale":"2023-12-15T23:00:50.369939+0000","last_undegraded":"2023-12-15T23:00:50.369939+0000","last_fullsized":"2023-12-15T23:00:50.369939+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"245'241","reported_seq":665,"reported_epoch":260,"state":"active+clean","last_fresh":"2023-12-15T23:00:48.649129+0000","last_change":"2023-12-15T23:00:48.655419+0000","last_active":"2023-12-15T23:00:48.649129+0000","last_peered":"2023-12-15T23:00:48.649129+0000","last_clean":"2023-12-15T23:00:48.649129+0000","last_became_active":"2023-12-15T22:56:39.506510+0000","last_became_peered":"2023-12-15T22:56:39.506510+0000","last_unstale":"2023-12-15T23:00:48.649129+0000","last_undegraded":"2023-12-15T23:00:48.649129+0000","last_fullsized":"2023-12-15T23:00:48.649129+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5496054,"num_objects":26,"num_object_clones":18,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":150,"num_read_kb":108658,"num_write":107,"num_write_kb":31471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"242'110","reported_seq":564,"reported_epoch":261,"state":"active+clean","last_fresh":"2023-12-15T23:00:49.321105+0000","last_change":"2023-12-15T23:00:48.323020+0000","last_active":"2023-12-15T23:00:49.321105+0000","last_peered":"2023-12-15T23:00:49.321105+0000","last_clean":"2023-12-15T23:00:49.321105+0000","last_became_active":"2023-12-15T22:56:39.511159+0000","last_became_peered":"2023-12-15T22:56:39.511159+0000","last_unstale":"2023-12-15T23:00:49.321105+0000","last_undegraded":"2023-12-15T23:00:49.321105+0000","last_fullsized":"2023-12-15T23:00:49.321105+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3932904,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":36,"num_read_kb":26874,"num_write":47,"num_write_kb":14375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":381,"reported_epoch":260,"state":"active+clean","last_fresh":"2023-12-15T23:00:48.646561+0000","last_change":"2023-12-15T22:56:33.633890+0000","last_active":"2023-12-15T23:00:48.646561+0000","last_peered":"2023-12-15T23:00:48.646561+0000","last_clean":"2023-12-15T23:00:48.646561+0000","last_became_active":"2023-12-15T22:56:33.631134+0000","last_became_peered":"2023-12-15T22:56:33.631134+0000","last_unstale":"2023-12-15T23:00:48.646561+0000","last_undegraded":"2023-12-15T23:00:48.646561+0000","last_fullsized":"2023-12-15T23:00:48.646561+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"245'173","reported_seq":611,"reported_epoch":260,"state":"active+clean","last_fresh":"2023-12-15T23:00:48.653033+0000","last_change":"2023-12-15T23:00:48.658260+0000","last_active":"2023-12-15T23:00:48.653033+0000","last_peered":"2023-12-15T23:00:48.653033+0000","last_clean":"2023-12-15T23:00:48.653033+0000","last_became_active":"2023-12-15T22:56:39.532730+0000","last_became_peered":"2023-12-15T22:56:39.532730+0000","last_unstale":"2023-12-15T23:00:48.653033+0000","last_undegraded":"2023-12-15T23:00:48.653033+0000","last_fullsized":"2023-12-15T23:00:48.653033+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":14,"num_object_clones":12,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":87,"num_read_kb":65976,"num_write":82,"num_write_kb":25453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":407,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-12-15T23:00:50.349073+0000","last_change":"2023-12-15T22:56:33.655736+0000","last_active":"2023-12-15T23:00:50.349073+0000","last_peered":"2023-12-15T23:00:50.349073+0000","last_clean":"2023-12-15T23:00:50.349073+0000","last_became_active":"2023-12-15T22:56:33.652917+0000","last_became_peered":"2023-12-15T22:56:33.652917+0000","last_unstale":"2023-12-15T23:00:50.349073+0000","last_undegraded":"2023-12-15T23:00:50.349073+0000","last_fullsized":"2023-12-15T23:00:50.349073+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"260'207","reported_seq":676,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-12-15T23:00:50.349724+0000","last_change":"2023-12-15T23:00:48.345310+0000","last_active":"2023-12-15T23:00:50.349724+0000","last_peered":"2023-12-15T23:00:50.349724+0000","last_clean":"2023-12-15T23:00:50.349724+0000","last_became_active":"2023-12-15T22:56:39.557252+0000","last_became_peered":"2023-12-15T22:56:39.557252+0000","last_unstale":"2023-12-15T23:00:50.349724+0000","last_undegraded":"2023-12-15T23:00:50.349724+0000","last_fullsized":"2023-12-15T23:00:50.349724+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6370018,"num_objects":19,"num_object_clones":13,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":127,"num_read_kb":96363,"num_write":98,"num_write_kb":27302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":381,"reported_epoch":260,"state":"active+clean","last_fresh":"2023-12-15T23:00:48.652225+0000","last_change":"2023-12-15T22:56:33.632348+0000","last_active":"2023-12-15T23:00:48.652225+0000","last_peered":"2023-12-15T23:00:48.652225+0000","last_clean":"2023-12-15T23:00:48.652225+0000","last_became_active":"2023-12-15T22:56:33.629022+0000","last_became_peered":"2023-12-15T22:56:33.629022+0000","last_unstale":"2023-12-15T23:00:48.652225+0000","last_undegraded":"2023-12-15T23:00:48.652225+0000","last_fullsized":"2023-12-15T23:00:48.652225+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"245'83","reported_seq":545,"reported_epoch":260,"state":"active+clean","last_fresh":"2023-12-15T23:00:48.654487+0000","last_change":"2023-12-15T23:00:48.658778+0000","last_active":"2023-12-15T23:00:48.654487+0000","last_peered":"2023-12-15T23:00:48.654487+0000","last_clean":"2023-12-15T23:00:48.654487+0000","last_became_active":"2023-12-15T22:56:39.535145+0000","last_became_peered":"2023-12-15T22:56:39.535145+0000","last_unstale":"2023-12-15T23:00:48.654487+0000","last_undegraded":"2023-12-15T23:00:48.654487+0000","last_fullsized":"2023-12-15T23:00:48.654487+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":6,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":29394,"num_write":35,"num_write_kb":10511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":393,"reported_epoch":261,"state":"active+clean","last_fresh":"2023-12-15T23:00:49.320636+0000","last_change":"2023-12-15T22:56:33.627005+0000","last_active":"2023-12-15T23:00:49.320636+0000","last_peered":"2023-12-15T23:00:49.320636+0000","last_clean":"2023-12-15T23:00:49.320636+0000","last_became_active":"2023-12-15T22:56:33.622181+0000","last_became_peered":"2023-12-15T22:56:33.622181+0000","last_unstale":"2023-12-15T23:00:49.320636+0000","last_undegraded":"2023-12-15T23:00:49.320636+0000","last_fullsized":"2023-12-15T23:00:49.320636+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":83432654,"num_objects":354,"num_object_clones":259,"num_object_copies":1062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":65,"num_read":2022,"num_read_kb":1518014,"num_write":1657,"num_write_kb":483419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3433,"ondisk_log_size":3433,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":94412,"kb_used_data":58392,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94277428,"statfs":{"total":96636764160,"available":96540086272,"internally_reserved":0,"allocated":59793408,"data_stored":64023498,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705923,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":77788,"kb_used_data":40648,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94294052,"statfs":{"total":96636764160,"available":96557109248,"internally_reserved":0,"allocated":41623552,"data_stored":44183106,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705923,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":101260,"kb_used_data":65116,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94270580,"statfs":{"total":96636764160,"available":96533073920,"internally_reserved":0,"allocated":66678784,"data_stored":70631217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":99196,"kb_used_data":63176,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94272644,"statfs":{"total":96636764160,"available":96535187456,"internally_reserved":0,"allocated":64692224,"data_stored":68876942,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-15T23:00:51.664 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T23:00:51.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:51.765+0000 7f321bade640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:51.772 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:51.772+0000 7f321bade640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:51.946 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T23:00:51.946 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-12-15T23:00:51.958 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":387,"stamp":"2023-12-15T23:00:51.315666+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":84023022,"num_objects":356,"num_object_clones":259,"num_object_copies":1068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":65,"num_read":2125,"num_read_kb":1518105,"num_write":1794,"num_write_kb":485313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3530,"ondisk_log_size":3530,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":372656,"kb_used_data":227332,"kb_used_omap":0,"kb_used_meta":140544,"kb_avail":377114704,"statfs":{"total":386547056640,"available":386165456896,"internally_reserved":0,"allocated":232787968,"data_stored":247714763,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":143917056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2185075,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":43758,"num_write":37,"num_write_kb":9971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.017137"},"pg_stats":[{"pgid":"3.f","version":"258'101","reported_seq":627,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-12-15T23:00:50.367191+0000","last_change":"2023-12-15T23:00:48.658797+0000","last_active":"2023-12-15T23:00:50.367191+0000","last_peered":"2023-12-15T23:00:50.367191+0000","last_clean":"2023-12-15T23:00:50.367191+0000","last_became_active":"2023-12-15T22:56:39.503797+0000","last_became_peered":"2023-12-15T22:56:39.503797+0000","last_unstale":"2023-12-15T23:00:50.367191+0000","last_undegraded":"2023-12-15T23:00:50.367191+0000","last_fullsized":"2023-12-15T23:00:50.367191+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6134687,"num_objects":12,"num_object_clones":8,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":104,"num_read_kb":90404,"num_write":44,"num_write_kb":13102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":496,"reported_epoch":260,"state":"active+clean","last_fresh":"2023-12-15T23:00:48.650884+0000","last_change":"2023-12-15T23:00:48.658556+0000","last_active":"2023-12-15T23:00:48.650884+0000","last_peered":"2023-12-15T23:00:48.650884+0000","last_clean":"2023-12-15T23:00:48.650884+0000","last_became_active":"2023-12-15T22:56:39.552856+0000","last_became_peered":"2023-12-15T22:56:39.552856+0000","last_unstale":"2023-12-15T23:00:48.650884+0000","last_undegraded":"2023-12-15T23:00:48.650884+0000","last_fullsized":"2023-12-15T23:00:48.650884+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"261'352","reported_seq":814,"reported_epoch":261,"state":"active+clean","last_fresh":"2023-12-15T23:00:49.469116+0000","last_change":"2023-12-15T23:00:48.344788+0000","last_active":"2023-12-15T23:00:49.469116+0000","last_peered":"2023-12-15T23:00:49.469116+0000","last_clean":"2023-12-15T23:00:49.469116+0000","last_became_active":"2023-12-15T22:56:39.495490+0000","last_became_peered":"2023-12-15T22:56:39.495490+0000","last_unstale":"2023-12-15T23:00:49.469116+0000","last_undegraded":"2023-12-15T23:00:49.469116+0000","last_fullsized":"2023-12-15T23:00:49.469116+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13513323,"num_objects":39,"num_object_clones":27,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":267,"num_read_kb":224831,"num_write":199,"num_write_kb":57479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"259'323","reported_seq":762,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-12-15T23:00:50.370390+0000","last_change":"2023-12-15T23:00:48.656617+0000","last_active":"2023-12-15T23:00:50.370390+0000","last_peered":"2023-12-15T23:00:50.370390+0000","last_clean":"2023-12-15T23:00:50.370390+0000","last_became_active":"2023-12-15T22:56:39.579234+0000","last_became_peered":"2023-12-15T22:56:39.579234+0000","last_unstale":"2023-12-15T23:00:50.370390+0000","last_undegraded":"2023-12-15T23:00:50.370390+0000","last_fullsized":"2023-12-15T23:00:50.370390+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2907710,"num_objects":31,"num_object_clones":28,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":162,"num_read_kb":112883,"num_write":167,"num_write_kb":48480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"260'183","reported_seq":641,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-12-15T23:00:50.347600+0000","last_change":"2023-12-15T23:00:48.322909+0000","last_active":"2023-12-15T23:00:50.347600+0000","last_peered":"2023-12-15T23:00:50.347600+0000","last_clean":"2023-12-15T23:00:50.347600+0000","last_became_active":"2023-12-15T22:56:39.572176+0000","last_became_peered":"2023-12-15T22:56:39.572176+0000","last_unstale":"2023-12-15T23:00:50.347600+0000","last_undegraded":"2023-12-15T23:00:50.347600+0000","last_fullsized":"2023-12-15T23:00:50.347600+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2941324,"num_objects":18,"num_object_clones":12,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":77,"num_read_kb":45677,"num_write":86,"num_write_kb":25133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"262'165","reported_seq":638,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-12-15T23:00:50.426673+0000","last_change":"2023-12-15T23:00:48.676042+0000","last_active":"2023-12-15T23:00:50.426673+0000","last_peered":"2023-12-15T23:00:50.426673+0000","last_clean":"2023-12-15T23:00:50.426673+0000","last_became_active":"2023-12-15T22:56:39.516361+0000","last_became_peered":"2023-12-15T22:56:39.516361+0000","last_unstale":"2023-12-15T23:00:50.426673+0000","last_undegraded":"2023-12-15T23:00:50.426673+0000","last_fullsized":"2023-12-15T23:00:50.426673+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5781806,"num_objects":17,"num_object_clones":12,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":74,"num_read_kb":56864,"num_write":73,"num_write_kb":21081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"260'634","reported_seq":920,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-12-15T23:00:50.338883+0000","last_change":"2023-12-15T23:00:48.678049+0000","last_active":"2023-12-15T23:00:50.338883+0000","last_peered":"2023-12-15T23:00:50.338883+0000","last_clean":"2023-12-15T23:00:50.338883+0000","last_became_active":"2023-12-15T22:56:39.519748+0000","last_became_peered":"2023-12-15T22:56:39.519748+0000","last_unstale":"2023-12-15T23:00:50.338883+0000","last_undegraded":"2023-12-15T23:00:50.338883+0000","last_fullsized":"2023-12-15T23:00:50.338883+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":634,"log_dups_size":0,"ondisk_log_size":634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14555291,"num_objects":65,"num_object_clones":50,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":282,"num_read_kb":177634,"num_write":274,"num_write_kb":78601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"256'161","reported_seq":661,"reported_epoch":261,"state":"active+clean","last_fresh":"2023-12-15T23:00:49.318640+0000","last_change":"2023-12-15T23:00:48.321538+0000","last_active":"2023-12-15T23:00:49.318640+0000","last_peered":"2023-12-15T23:00:49.318640+0000","last_clean":"2023-12-15T23:00:49.318640+0000","last_became_active":"2023-12-15T22:56:39.543179+0000","last_became_peered":"2023-12-15T22:56:39.543179+0000","last_unstale":"2023-12-15T23:00:49.318640+0000","last_undegraded":"2023-12-15T23:00:49.318640+0000","last_fullsized":"2023-12-15T23:00:49.318640+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7260888,"num_objects":18,"num_object_clones":12,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":155,"num_read_kb":134340,"num_write":91,"num_write_kb":25030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":536,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-12-15T23:00:50.349006+0000","last_change":"2023-12-15T22:56:32.607017+0000","last_active":"2023-12-15T23:00:50.349006+0000","last_peered":"2023-12-15T23:00:50.349006+0000","last_clean":"2023-12-15T23:00:50.349006+0000","last_became_active":"2023-12-15T22:56:32.602384+0000","last_became_peered":"2023-12-15T22:56:32.602384+0000","last_unstale":"2023-12-15T23:00:50.349006+0000","last_undegraded":"2023-12-15T23:00:50.349006+0000","last_fullsized":"2023-12-15T23:00:50.349006+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_clean_scrub_stamp":"2023-12-15T22:56:31.525572+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":393,"reported_epoch":261,"state":"active+clean","last_fresh":"2023-12-15T23:00:49.320577+0000","last_change":"2023-12-15T22:56:33.627481+0000","last_active":"2023-12-15T23:00:49.320577+0000","last_peered":"2023-12-15T23:00:49.320577+0000","last_clean":"2023-12-15T23:00:49.320577+0000","last_became_active":"2023-12-15T22:56:33.624689+0000","last_became_peered":"2023-12-15T22:56:33.624689+0000","last_unstale":"2023-12-15T23:00:49.320577+0000","last_undegraded":"2023-12-15T23:00:49.320577+0000","last_fullsized":"2023-12-15T23:00:49.320577+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"251'248","reported_seq":742,"reported_epoch":261,"state":"active+clean","last_fresh":"2023-12-15T23:00:49.317617+0000","last_change":"2023-12-15T23:00:48.318145+0000","last_active":"2023-12-15T23:00:49.317617+0000","last_peered":"2023-12-15T23:00:49.317617+0000","last_clean":"2023-12-15T23:00:49.317617+0000","last_became_active":"2023-12-15T22:56:39.529359+0000","last_became_peered":"2023-12-15T22:56:39.529359+0000","last_unstale":"2023-12-15T23:00:49.317617+0000","last_undegraded":"2023-12-15T23:00:49.317617+0000","last_fullsized":"2023-12-15T23:00:49.317617+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756579,"num_objects":30,"num_object_clones":19,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":205,"num_read_kb":148054,"num_write":149,"num_write_kb":43104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":425,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-12-15T23:00:50.356183+0000","last_change":"2023-12-15T22:56:33.655797+0000","last_active":"2023-12-15T23:00:50.356183+0000","last_peered":"2023-12-15T23:00:50.356183+0000","last_clean":"2023-12-15T23:00:50.356183+0000","last_became_active":"2023-12-15T22:56:33.652885+0000","last_became_peered":"2023-12-15T22:56:33.652885+0000","last_unstale":"2023-12-15T23:00:50.356183+0000","last_undegraded":"2023-12-15T23:00:50.356183+0000","last_fullsized":"2023-12-15T23:00:50.356183+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"260'306","reported_seq":732,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-12-15T23:00:50.340483+0000","last_change":"2023-12-15T23:00:48.327524+0000","last_active":"2023-12-15T23:00:50.340483+0000","last_peered":"2023-12-15T23:00:50.340483+0000","last_clean":"2023-12-15T23:00:50.340483+0000","last_became_active":"2023-12-15T22:56:39.543509+0000","last_became_peered":"2023-12-15T22:56:39.543509+0000","last_unstale":"2023-12-15T23:00:50.340483+0000","last_undegraded":"2023-12-15T23:00:50.340483+0000","last_fullsized":"2023-12-15T23:00:50.340483+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":306,"log_dups_size":0,"ondisk_log_size":306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4554895,"num_objects":35,"num_object_clones":24,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":180,"num_read_kb":134405,"num_write":135,"num_write_kb":41661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":393,"reported_epoch":261,"state":"active+clean","last_fresh":"2023-12-15T23:00:49.318036+0000","last_change":"2023-12-15T22:56:33.600485+0000","last_active":"2023-12-15T23:00:49.318036+0000","last_peered":"2023-12-15T23:00:49.318036+0000","last_clean":"2023-12-15T23:00:49.318036+0000","last_became_active":"2023-12-15T22:56:33.597320+0000","last_became_peered":"2023-12-15T22:56:33.597320+0000","last_unstale":"2023-12-15T23:00:49.318036+0000","last_undegraded":"2023-12-15T23:00:49.318036+0000","last_fullsized":"2023-12-15T23:00:49.318036+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"260'146","reported_seq":622,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-12-15T23:00:50.347753+0000","last_change":"2023-12-15T23:00:48.343368+0000","last_active":"2023-12-15T23:00:50.347753+0000","last_peered":"2023-12-15T23:00:50.347753+0000","last_clean":"2023-12-15T23:00:50.347753+0000","last_became_active":"2023-12-15T22:56:39.560980+0000","last_became_peered":"2023-12-15T22:56:39.560980+0000","last_unstale":"2023-12-15T23:00:50.347753+0000","last_undegraded":"2023-12-15T23:00:50.347753+0000","last_fullsized":"2023-12-15T23:00:50.347753+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5227175,"num_objects":13,"num_object_clones":10,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":81,"num_read_kb":65657,"num_write":70,"num_write_kb":20636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":425,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-12-15T23:00:50.369939+0000","last_change":"2023-12-15T22:56:33.648541+0000","last_active":"2023-12-15T23:00:50.369939+0000","last_peered":"2023-12-15T23:00:50.369939+0000","last_clean":"2023-12-15T23:00:50.369939+0000","last_became_active":"2023-12-15T22:56:33.643681+0000","last_became_peered":"2023-12-15T22:56:33.643681+0000","last_unstale":"2023-12-15T23:00:50.369939+0000","last_undegraded":"2023-12-15T23:00:50.369939+0000","last_fullsized":"2023-12-15T23:00:50.369939+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"245'241","reported_seq":665,"reported_epoch":260,"state":"active+clean","last_fresh":"2023-12-15T23:00:48.649129+0000","last_change":"2023-12-15T23:00:48.655419+0000","last_active":"2023-12-15T23:00:48.649129+0000","last_peered":"2023-12-15T23:00:48.649129+0000","last_clean":"2023-12-15T23:00:48.649129+0000","last_became_active":"2023-12-15T22:56:39.506510+0000","last_became_peered":"2023-12-15T22:56:39.506510+0000","last_unstale":"2023-12-15T23:00:48.649129+0000","last_undegraded":"2023-12-15T23:00:48.649129+0000","last_fullsized":"2023-12-15T23:00:48.649129+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5496054,"num_objects":26,"num_object_clones":18,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":150,"num_read_kb":108658,"num_write":107,"num_write_kb":31471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"242'110","reported_seq":564,"reported_epoch":261,"state":"active+clean","last_fresh":"2023-12-15T23:00:49.321105+0000","last_change":"2023-12-15T23:00:48.323020+0000","last_active":"2023-12-15T23:00:49.321105+0000","last_peered":"2023-12-15T23:00:49.321105+0000","last_clean":"2023-12-15T23:00:49.321105+0000","last_became_active":"2023-12-15T22:56:39.511159+0000","last_became_peered":"2023-12-15T22:56:39.511159+0000","last_unstale":"2023-12-15T23:00:49.321105+0000","last_undegraded":"2023-12-15T23:00:49.321105+0000","last_fullsized":"2023-12-15T23:00:49.321105+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3932904,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":36,"num_read_kb":26874,"num_write":47,"num_write_kb":14375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":381,"reported_epoch":260,"state":"active+clean","last_fresh":"2023-12-15T23:00:48.646561+0000","last_change":"2023-12-15T22:56:33.633890+0000","last_active":"2023-12-15T23:00:48.646561+0000","last_peered":"2023-12-15T23:00:48.646561+0000","last_clean":"2023-12-15T23:00:48.646561+0000","last_became_active":"2023-12-15T22:56:33.631134+0000","last_became_peered":"2023-12-15T22:56:33.631134+0000","last_unstale":"2023-12-15T23:00:48.646561+0000","last_undegraded":"2023-12-15T23:00:48.646561+0000","last_fullsized":"2023-12-15T23:00:48.646561+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"245'173","reported_seq":611,"reported_epoch":260,"state":"active+clean","last_fresh":"2023-12-15T23:00:48.653033+0000","last_change":"2023-12-15T23:00:48.658260+0000","last_active":"2023-12-15T23:00:48.653033+0000","last_peered":"2023-12-15T23:00:48.653033+0000","last_clean":"2023-12-15T23:00:48.653033+0000","last_became_active":"2023-12-15T22:56:39.532730+0000","last_became_peered":"2023-12-15T22:56:39.532730+0000","last_unstale":"2023-12-15T23:00:48.653033+0000","last_undegraded":"2023-12-15T23:00:48.653033+0000","last_fullsized":"2023-12-15T23:00:48.653033+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":14,"num_object_clones":12,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":87,"num_read_kb":65976,"num_write":82,"num_write_kb":25453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":407,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-12-15T23:00:50.349073+0000","last_change":"2023-12-15T22:56:33.655736+0000","last_active":"2023-12-15T23:00:50.349073+0000","last_peered":"2023-12-15T23:00:50.349073+0000","last_clean":"2023-12-15T23:00:50.349073+0000","last_became_active":"2023-12-15T22:56:33.652917+0000","last_became_peered":"2023-12-15T22:56:33.652917+0000","last_unstale":"2023-12-15T23:00:50.349073+0000","last_undegraded":"2023-12-15T23:00:50.349073+0000","last_fullsized":"2023-12-15T23:00:50.349073+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"260'207","reported_seq":676,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-12-15T23:00:50.349724+0000","last_change":"2023-12-15T23:00:48.345310+0000","last_active":"2023-12-15T23:00:50.349724+0000","last_peered":"2023-12-15T23:00:50.349724+0000","last_clean":"2023-12-15T23:00:50.349724+0000","last_became_active":"2023-12-15T22:56:39.557252+0000","last_became_peered":"2023-12-15T22:56:39.557252+0000","last_unstale":"2023-12-15T23:00:50.349724+0000","last_undegraded":"2023-12-15T23:00:50.349724+0000","last_fullsized":"2023-12-15T23:00:50.349724+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6370018,"num_objects":19,"num_object_clones":13,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":127,"num_read_kb":96363,"num_write":98,"num_write_kb":27302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":381,"reported_epoch":260,"state":"active+clean","last_fresh":"2023-12-15T23:00:48.652225+0000","last_change":"2023-12-15T22:56:33.632348+0000","last_active":"2023-12-15T23:00:48.652225+0000","last_peered":"2023-12-15T23:00:48.652225+0000","last_clean":"2023-12-15T23:00:48.652225+0000","last_became_active":"2023-12-15T22:56:33.629022+0000","last_became_peered":"2023-12-15T22:56:33.629022+0000","last_unstale":"2023-12-15T23:00:48.652225+0000","last_undegraded":"2023-12-15T23:00:48.652225+0000","last_fullsized":"2023-12-15T23:00:48.652225+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"245'83","reported_seq":545,"reported_epoch":260,"state":"active+clean","last_fresh":"2023-12-15T23:00:48.654487+0000","last_change":"2023-12-15T23:00:48.658778+0000","last_active":"2023-12-15T23:00:48.654487+0000","last_peered":"2023-12-15T23:00:48.654487+0000","last_clean":"2023-12-15T23:00:48.654487+0000","last_became_active":"2023-12-15T22:56:39.535145+0000","last_became_peered":"2023-12-15T22:56:39.535145+0000","last_unstale":"2023-12-15T23:00:48.654487+0000","last_undegraded":"2023-12-15T23:00:48.654487+0000","last_fullsized":"2023-12-15T23:00:48.654487+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":6,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":29394,"num_write":35,"num_write_kb":10511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":393,"reported_epoch":261,"state":"active+clean","last_fresh":"2023-12-15T23:00:49.320636+0000","last_change":"2023-12-15T22:56:33.627005+0000","last_active":"2023-12-15T23:00:49.320636+0000","last_peered":"2023-12-15T23:00:49.320636+0000","last_clean":"2023-12-15T23:00:49.320636+0000","last_became_active":"2023-12-15T22:56:33.622181+0000","last_became_peered":"2023-12-15T22:56:33.622181+0000","last_unstale":"2023-12-15T23:00:49.320636+0000","last_undegraded":"2023-12-15T23:00:49.320636+0000","last_fullsized":"2023-12-15T23:00:49.320636+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":83432654,"num_objects":354,"num_object_clones":259,"num_object_copies":1062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":65,"num_read":2022,"num_read_kb":1518014,"num_write":1657,"num_write_kb":483419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3433,"ondisk_log_size":3433,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":94412,"kb_used_data":58392,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94277428,"statfs":{"total":96636764160,"available":96540086272,"internally_reserved":0,"allocated":59793408,"data_stored":64023498,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705923,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":77788,"kb_used_data":40648,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94294052,"statfs":{"total":96636764160,"available":96557109248,"internally_reserved":0,"allocated":41623552,"data_stored":44183106,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705923,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":101260,"kb_used_data":65116,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94270580,"statfs":{"total":96636764160,"available":96533073920,"internally_reserved":0,"allocated":66678784,"data_stored":70631217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":99196,"kb_used_data":63176,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94272644,"statfs":{"total":96636764160,"available":96535187456,"internally_reserved":0,"allocated":64692224,"data_stored":68876942,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-15T23:00:51.959 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2023-12-15T23:00:51.959 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T23:00:51.959 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-12-15T23:00:51.959 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-12-15T23:00:52.066 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:52.061+0000 7f3d1b26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:52.068 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T23:00:52.069 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:52.069+0000 7f3d1b26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:52.081 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T23:00:52.082 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T23:00:52.082 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T23:00:52.082 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T23:00:52.082 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T23:00:52.082 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T23:00:52.082 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T23:00:52.082 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T23:00:52.082 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T23:00:52.082 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T23:00:52.082 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T23:00:52.082 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T23:00:52.085 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T23:00:52.085 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:00:52.085 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-12-15T23:00:52.173 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:52.172+0000 7fba4a0f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:52.176 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:52.176+0000 7fba4a0f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:52.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:52.244+0000 7fdf4c8de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:52.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:52.251+0000 7fdf4c8de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:52.265 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T23:00:52.265 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T23:00:52.265 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T23:00:52.265 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T23:00:52.265 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T23:00:52.265 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T23:00:52.265 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T23:00:52.265 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T23:00:52.266 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T23:00:52.266 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T23:00:52.266 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T23:00:52.266 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T23:00:52.269 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T23:00:52.269 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:00:52.269 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-12-15T23:00:52.284 INFO:tasks.rados.rados.0.smithi071.stdout:1104: done (0 left) 2023-12-15T23:00:52.284 INFO:tasks.rados.rados.0.smithi071.stdout:1105: snap_create 2023-12-15T23:00:52.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:52.377+0000 7f0c7062d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:52.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:52.384+0000 7f0c7062d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:52.397 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T23:00:52.397 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T23:00:52.397 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T23:00:52.397 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T23:00:52.398 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T23:00:52.398 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T23:00:52.398 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T23:00:52.398 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T23:00:52.398 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T23:00:52.398 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T23:00:52.398 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T23:00:52.398 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T23:00:52.400 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T23:00:52.400 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:00:52.400 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-12-15T23:00:52.559 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:52.554+0000 7f53f66ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:52.562 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:52.562+0000 7f53f66ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:52.665 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T23:00:52.665 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T23:00:52.665 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T23:00:52.665 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T23:00:52.665 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T23:00:52.665 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T23:00:52.665 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T23:00:52.665 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T23:00:52.665 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T23:00:52.666 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T23:00:52.666 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T23:00:52.666 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T23:00:52.668 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T23:00:52.668 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:00:53.289 INFO:tasks.rados.rados.0.smithi071.stdout:1105: done (0 left) 2023-12-15T23:00:53.290 INFO:tasks.rados.rados.0.smithi071.stdout:1106: write (excl) oid 40 current snap is 124 2023-12-15T23:00:53.290 INFO:tasks.rados.rados.0.smithi071.stdout:1106: seq_num 537 ranges {625359=553423,1790948=567184,2930500=94859} 2023-12-15T23:00:53.296 INFO:tasks.rados.rados.0.smithi071.stdout:1106: writing smithi07183189-40 from 625359 to 1178782 tid 1 2023-12-15T23:00:53.302 INFO:tasks.rados.rados.0.smithi071.stdout:1106: writing smithi07183189-40 from 1790948 to 2358132 tid 2 2023-12-15T23:00:53.306 INFO:tasks.rados.rados.0.smithi071.stdout:1106: writing smithi07183189-40 from 2930500 to 3025359 tid 3 2023-12-15T23:00:53.306 INFO:tasks.rados.rados.0.smithi071.stdout:1107: write (excl) oid 48 current snap is 124 2023-12-15T23:00:53.306 INFO:tasks.rados.rados.0.smithi071.stdout:1107: seq_num 538 ranges {642166=628672,1849430=710842} 2023-12-15T23:00:53.314 INFO:tasks.rados.rados.0.smithi071.stdout:1107: writing smithi07183189-48 from 642166 to 1270838 tid 1 2023-12-15T23:00:53.323 INFO:tasks.rados.rados.0.smithi071.stdout:1107: writing smithi07183189-48 from 1849430 to 2560272 tid 2 2023-12-15T23:00:53.323 INFO:tasks.rados.rados.0.smithi071.stdout:1108: write (excl) oid 13 current snap is 124 2023-12-15T23:00:53.324 INFO:tasks.rados.rados.0.smithi071.stdout:1108: seq_num 539 ranges {658973=703921,1907912=570853,2985346=73627} 2023-12-15T23:00:53.333 INFO:tasks.rados.rados.0.smithi071.stdout:1108: writing smithi07183189-13 from 658973 to 1362894 tid 1 2023-12-15T23:00:53.341 INFO:tasks.rados.rados.0.smithi071.stdout:1108: writing smithi07183189-13 from 1907912 to 2478765 tid 2 2023-12-15T23:00:53.345 INFO:tasks.rados.rados.0.smithi071.stdout:1108: writing smithi07183189-13 from 2985346 to 3058973 tid 3 2023-12-15T23:00:53.345 INFO:tasks.rados.rados.0.smithi071.stdout:1109: write (excl) oid 43 current snap is 124 2023-12-15T23:00:53.345 INFO:tasks.rados.rados.0.smithi071.stdout:1109: seq_num 540 ranges {675780=495523,1682747=714511,3012769=63011} 2023-12-15T23:00:53.352 INFO:tasks.rados.rados.0.smithi071.stdout:1109: writing smithi07183189-43 from 675780 to 1171303 tid 1 2023-12-15T23:00:53.361 INFO:tasks.rados.rados.0.smithi071.stdout:1109: writing smithi07183189-43 from 1682747 to 2397258 tid 2 2023-12-15T23:00:53.365 INFO:tasks.rados.rados.0.smithi071.stdout:1109: writing smithi07183189-43 from 3012769 to 3075780 tid 3 2023-12-15T23:00:53.366 INFO:tasks.rados.rados.0.smithi071.stdout:1110: snap_remove snap 120 2023-12-15T23:00:54.269 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T23:00:54.331 INFO:tasks.rados.rados.0.smithi071.stdout:1110: done (4 left) 2023-12-15T23:00:54.331 INFO:tasks.rados.rados.0.smithi071.stdout:1111: write oid 3 current snap is 124 2023-12-15T23:00:54.331 INFO:tasks.rados.rados.0.smithi071.stdout:1106: finishing write tid 1 to smithi07183189-40 2023-12-15T23:00:54.331 INFO:tasks.rados.rados.0.smithi071.stdout:1111: seq_num 541 ranges {692587=570772,2024876=574522,3040192=52395} 2023-12-15T23:00:54.338 INFO:tasks.rados.rados.0.smithi071.stdout:1111: writing smithi07183189-3 from 692587 to 1263359 tid 1 2023-12-15T23:00:54.346 INFO:tasks.rados.rados.0.smithi071.stdout:1111: writing smithi07183189-3 from 2024876 to 2599398 tid 2 2023-12-15T23:00:54.349 INFO:tasks.rados.rados.0.smithi071.stdout:1111: writing smithi07183189-3 from 3040192 to 3092587 tid 3 2023-12-15T23:00:54.350 INFO:tasks.rados.rados.0.smithi071.stdout:1112: snap_create 2023-12-15T23:00:54.350 INFO:tasks.rados.rados.0.smithi071.stdout:1106: finishing write tid 2 to smithi07183189-40 2023-12-15T23:00:54.350 INFO:tasks.rados.rados.0.smithi071.stdout:1108: finishing write tid 1 to smithi07183189-13 2023-12-15T23:00:54.350 INFO:tasks.rados.rados.0.smithi071.stdout:1107: finishing write tid 1 to smithi07183189-48 2023-12-15T23:00:54.350 INFO:tasks.rados.rados.0.smithi071.stdout:1109: finishing write tid 1 to smithi07183189-43 2023-12-15T23:00:54.350 INFO:tasks.rados.rados.0.smithi071.stdout:1106: finishing write tid 3 to smithi07183189-40 2023-12-15T23:00:54.350 INFO:tasks.rados.rados.0.smithi071.stdout:1108: finishing write tid 2 to smithi07183189-13 2023-12-15T23:00:54.350 INFO:tasks.rados.rados.0.smithi071.stdout:1106: finishing write tid 4 to smithi07183189-40 2023-12-15T23:00:54.350 INFO:tasks.rados.rados.0.smithi071.stdout:1106: finishing write tid 5 to smithi07183189-40 2023-12-15T23:00:54.350 INFO:tasks.rados.rados.0.smithi071.stdout:1106: oid 40 updating version 0 to 641 2023-12-15T23:00:54.350 INFO:tasks.rados.rados.0.smithi071.stdout:1106: oid 40 updating version 641 to 643 2023-12-15T23:00:54.350 INFO:tasks.rados.rados.0.smithi071.stdout:1106: oid 40 version 643 is already newer than 640 2023-12-15T23:00:54.350 INFO:tasks.rados.rados.0.smithi071.stdout:1106: oid 40 version 643 is already newer than 642 2023-12-15T23:00:54.350 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 40 v 643 (ObjNum 537 snap 124 seq_num 537) dirty exists 2023-12-15T23:00:54.351 INFO:tasks.rados.rados.0.smithi071.stdout:1106: left oid 40 (ObjNum 537 snap 124 seq_num 537) 2023-12-15T23:00:54.351 INFO:tasks.rados.rados.0.smithi071.stdout:1109: finishing write tid 2 to smithi07183189-43 2023-12-15T23:00:54.351 INFO:tasks.rados.rados.0.smithi071.stdout:1107: finishing write tid 2 to smithi07183189-48 2023-12-15T23:00:54.351 INFO:tasks.rados.rados.0.smithi071.stdout:1108: finishing write tid 3 to smithi07183189-13 2023-12-15T23:00:54.351 INFO:tasks.rados.rados.0.smithi071.stdout:1108: finishing write tid 4 to smithi07183189-13 2023-12-15T23:00:54.351 INFO:tasks.rados.rados.0.smithi071.stdout:1108: finishing write tid 5 to smithi07183189-13 2023-12-15T23:00:54.351 INFO:tasks.rados.rados.0.smithi071.stdout:1108: oid 13 updating version 0 to 316 2023-12-15T23:00:54.351 INFO:tasks.rados.rados.0.smithi071.stdout:1108: oid 13 version 316 is already newer than 314 2023-12-15T23:00:54.351 INFO:tasks.rados.rados.0.smithi071.stdout:1108: oid 13 version 316 is already newer than 316 2023-12-15T23:00:54.351 INFO:tasks.rados.rados.0.smithi071.stdout:1108: oid 13 version 316 is already newer than 313 2023-12-15T23:00:54.351 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 13 v 316 (ObjNum 539 snap 124 seq_num 539) dirty exists 2023-12-15T23:00:54.351 INFO:tasks.rados.rados.0.smithi071.stdout:1108: left oid 13 (ObjNum 539 snap 124 seq_num 539) 2023-12-15T23:00:54.351 INFO:tasks.rados.rados.0.smithi071.stdout:1109: finishing write tid 3 to smithi07183189-43 2023-12-15T23:00:54.351 INFO:tasks.rados.rados.0.smithi071.stdout:1107: finishing write tid 3 to smithi07183189-48 2023-12-15T23:00:54.351 INFO:tasks.rados.rados.0.smithi071.stdout:1107: finishing write tid 4 to smithi07183189-48 2023-12-15T23:00:54.351 INFO:tasks.rados.rados.0.smithi071.stdout:1107: oid 48 updating version 0 to 251 2023-12-15T23:00:54.351 INFO:tasks.rados.rados.0.smithi071.stdout:1107: oid 48 version 251 is already newer than 250 2023-12-15T23:00:54.352 INFO:tasks.rados.rados.0.smithi071.stdout:1107: oid 48 updating version 251 to 252 2023-12-15T23:00:54.352 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 252 (ObjNum 538 snap 124 seq_num 538) dirty exists 2023-12-15T23:00:54.352 INFO:tasks.rados.rados.0.smithi071.stdout:1107: left oid 48 (ObjNum 538 snap 124 seq_num 538) 2023-12-15T23:00:54.352 INFO:tasks.rados.rados.0.smithi071.stdout:1109: finishing write tid 4 to smithi07183189-43 2023-12-15T23:00:54.352 INFO:tasks.rados.rados.0.smithi071.stdout:1109: finishing write tid 5 to smithi07183189-43 2023-12-15T23:00:54.352 INFO:tasks.rados.rados.0.smithi071.stdout:1109: oid 43 updating version 0 to 150 2023-12-15T23:00:54.352 INFO:tasks.rados.rados.0.smithi071.stdout:1109: oid 43 updating version 150 to 152 2023-12-15T23:00:54.352 INFO:tasks.rados.rados.0.smithi071.stdout:1109: oid 43 updating version 152 to 153 2023-12-15T23:00:54.352 INFO:tasks.rados.rados.0.smithi071.stdout:1109: oid 43 version 153 is already newer than 151 2023-12-15T23:00:54.352 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 43 v 153 (ObjNum 540 snap 124 seq_num 540) dirty exists 2023-12-15T23:00:54.352 INFO:tasks.rados.rados.0.smithi071.stdout:1109: left oid 43 (ObjNum 540 snap 124 seq_num 540) 2023-12-15T23:00:54.410 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:54.409+0000 7f04f202d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:54.414 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:54.414+0000 7f04f202d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:54.457 INFO:tasks.rados.rados.0.smithi071.stdout:1111: finishing write tid 1 to smithi07183189-3 2023-12-15T23:00:54.485 INFO:tasks.rados.rados.0.smithi071.stdout:1111: finishing write tid 2 to smithi07183189-3 2023-12-15T23:00:54.487 INFO:tasks.rados.rados.0.smithi071.stdout:1111: finishing write tid 3 to smithi07183189-3 2023-12-15T23:00:54.500 INFO:tasks.rados.rados.0.smithi071.stdout:1111: finishing write tid 4 to smithi07183189-3 2023-12-15T23:00:54.501 INFO:tasks.rados.rados.0.smithi071.stdout:1111: finishing write tid 5 to smithi07183189-3 2023-12-15T23:00:54.501 INFO:tasks.rados.rados.0.smithi071.stdout:1111: oid 3 updating version 0 to 168 2023-12-15T23:00:54.501 INFO:tasks.rados.rados.0.smithi071.stdout:1111: oid 3 version 168 is already newer than 165 2023-12-15T23:00:54.501 INFO:tasks.rados.rados.0.smithi071.stdout:1111: oid 3 version 168 is already newer than 167 2023-12-15T23:00:54.501 INFO:tasks.rados.rados.0.smithi071.stdout:1111: oid 3 version 168 is already newer than 167 2023-12-15T23:00:54.501 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 3 v 168 (ObjNum 541 snap 124 seq_num 541) dirty exists 2023-12-15T23:00:54.501 INFO:tasks.rados.rados.0.smithi071.stdout:1111: left oid 3 (ObjNum 541 snap 124 seq_num 541) 2023-12-15T23:00:55.334 INFO:tasks.rados.rados.0.smithi071.stdout:1106: done (5 left) 2023-12-15T23:00:55.335 INFO:tasks.rados.rados.0.smithi071.stdout:1107: done (4 left) 2023-12-15T23:00:55.335 INFO:tasks.rados.rados.0.smithi071.stdout:1108: done (3 left) 2023-12-15T23:00:55.335 INFO:tasks.rados.rados.0.smithi071.stdout:1109: done (2 left) 2023-12-15T23:00:55.335 INFO:tasks.rados.rados.0.smithi071.stdout:1111: done (1 left) 2023-12-15T23:00:55.335 INFO:tasks.rados.rados.0.smithi071.stdout:1112: done (0 left) 2023-12-15T23:00:55.335 INFO:tasks.rados.rados.0.smithi071.stdout:1113: snap_create 2023-12-15T23:00:56.336 INFO:tasks.rados.rados.0.smithi071.stdout:1113: done (0 left) 2023-12-15T23:00:56.336 INFO:tasks.rados.rados.0.smithi071.stdout:1114: read oid 17 snap -1 2023-12-15T23:00:56.336 INFO:tasks.rados.rados.0.smithi071.stdout:1114: expect deleted 2023-12-15T23:00:56.336 INFO:tasks.rados.rados.0.smithi071.stdout:1115: write oid 12 current snap is 126 2023-12-15T23:00:56.336 INFO:tasks.rados.rados.0.smithi071.stdout:1115: seq_num 542 ranges {709394=646021,2083358=718180} 2023-12-15T23:00:56.343 INFO:tasks.rados.rados.0.smithi071.stdout:1115: writing smithi07183189-12 from 709394 to 1355415 tid 1 2023-12-15T23:00:56.351 INFO:tasks.rados.rados.0.smithi071.stdout:1115: writing smithi07183189-12 from 2083358 to 2801538 tid 2 2023-12-15T23:00:56.351 INFO:tasks.rados.rados.0.smithi071.stdout:1116: read oid 7 snap -1 2023-12-15T23:00:56.351 INFO:tasks.rados.rados.0.smithi071.stdout:1116: expect (ObjNum 533 snap 122 seq_num 533) 2023-12-15T23:00:56.351 INFO:tasks.rados.rados.0.smithi071.stdout:1117: snap_remove snap 124 2023-12-15T23:00:56.538 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T23:00:56.653 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:56.652+0000 7f7755b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:56.657 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:56.657+0000 7f7755b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:57.342 INFO:tasks.rados.rados.0.smithi071.stdout:1117: done (3 left) 2023-12-15T23:00:57.342 INFO:tasks.rados.rados.0.smithi071.stdout:1118: snap_create 2023-12-15T23:00:57.342 INFO:tasks.rados.rados.0.smithi071.stdout:1116: expect (ObjNum 533 snap 122 seq_num 533) 2023-12-15T23:00:57.495 INFO:tasks.rados.rados.0.smithi071.stdout:1115: finishing write tid 1 to smithi07183189-12 2023-12-15T23:00:57.495 INFO:tasks.rados.rados.0.smithi071.stdout:1115: finishing write tid 2 to smithi07183189-12 2023-12-15T23:00:57.495 INFO:tasks.rados.rados.0.smithi071.stdout:1115: finishing write tid 3 to smithi07183189-12 2023-12-15T23:00:57.495 INFO:tasks.rados.rados.0.smithi071.stdout:1115: finishing write tid 4 to smithi07183189-12 2023-12-15T23:00:57.495 INFO:tasks.rados.rados.0.smithi071.stdout:1115: oid 12 updating version 0 to 186 2023-12-15T23:00:57.495 INFO:tasks.rados.rados.0.smithi071.stdout:1115: oid 12 version 186 is already newer than 184 2023-12-15T23:00:57.495 INFO:tasks.rados.rados.0.smithi071.stdout:1115: oid 12 version 186 is already newer than 185 2023-12-15T23:00:57.495 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 12 v 186 (ObjNum 542 snap 126 seq_num 542) dirty exists 2023-12-15T23:00:57.495 INFO:tasks.rados.rados.0.smithi071.stdout:1115: left oid 12 (ObjNum 542 snap 126 seq_num 542) 2023-12-15T23:00:57.669 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T23:00:57.670 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T23:00:57.670 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-12-15T23:00:57.670 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-12-15T23:00:57.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:57.775+0000 7f4b5b99e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:57.783 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:57.783+0000 7f4b5b99e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:57.795 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T23:00:57.795 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T23:00:57.795 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T23:00:57.795 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T23:00:57.795 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T23:00:57.795 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T23:00:57.795 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T23:00:57.795 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T23:00:57.795 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T23:00:57.795 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T23:00:57.795 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T23:00:57.795 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T23:00:57.797 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T23:00:57.798 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:00:57.798 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-12-15T23:00:57.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:57.900+0000 7f38c839e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:57.908 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:57.908+0000 7f38c839e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:57.920 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T23:00:57.920 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T23:00:57.920 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T23:00:57.920 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T23:00:57.920 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T23:00:57.920 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T23:00:57.920 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T23:00:57.920 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T23:00:57.920 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T23:00:57.920 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T23:00:57.920 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T23:00:57.920 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T23:00:57.922 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T23:00:57.923 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:00:57.923 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-12-15T23:00:58.030 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:58.025+0000 7f0ba8d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:58.033 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:58.033+0000 7f0ba8d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:58.046 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T23:00:58.046 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T23:00:58.046 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T23:00:58.046 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T23:00:58.046 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T23:00:58.046 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T23:00:58.046 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T23:00:58.046 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T23:00:58.046 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T23:00:58.046 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T23:00:58.046 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T23:00:58.047 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T23:00:58.048 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T23:00:58.049 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:00:58.049 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-12-15T23:00:58.155 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:58.150+0000 7feccc2f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:58.158 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:58.158+0000 7feccc2f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:58.171 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T23:00:58.171 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T23:00:58.171 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T23:00:58.171 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T23:00:58.171 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T23:00:58.171 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T23:00:58.171 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T23:00:58.172 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T23:00:58.172 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T23:00:58.172 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T23:00:58.172 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T23:00:58.172 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T23:00:58.173 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T23:00:58.174 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:00:58.343 INFO:tasks.rados.rados.0.smithi071.stdout:1114: done (3 left) 2023-12-15T23:00:58.343 INFO:tasks.rados.rados.0.smithi071.stdout:1115: done (2 left) 2023-12-15T23:00:58.343 INFO:tasks.rados.rados.0.smithi071.stdout:1116: done (1 left) 2023-12-15T23:00:58.344 INFO:tasks.rados.rados.0.smithi071.stdout:1118: done (0 left) 2023-12-15T23:00:58.344 INFO:tasks.rados.rados.0.smithi071.stdout:1119: write oid 1 current snap is 127 2023-12-15T23:00:58.344 INFO:tasks.rados.rados.0.smithi071.stdout:1119: seq_num 543 ranges {726201=721270,2141840=578191} 2023-12-15T23:00:58.351 INFO:tasks.rados.rados.0.smithi071.stdout:1119: writing smithi07183189-1 from 726201 to 1447471 tid 1 2023-12-15T23:00:58.358 INFO:tasks.rados.rados.0.smithi071.stdout:1119: writing smithi07183189-1 from 2141840 to 2720031 tid 2 2023-12-15T23:00:58.359 INFO:tasks.rados.rados.0.smithi071.stdout:1120: delete oid 30 current snap is 127 2023-12-15T23:00:58.416 INFO:tasks.rados.rados.0.smithi071.stdout:1120: done (1 left) 2023-12-15T23:00:58.416 INFO:tasks.rados.rados.0.smithi071.stdout:1121: read oid 24 snap -1 2023-12-15T23:00:58.416 INFO:tasks.rados.rados.0.smithi071.stdout:1121: expect (ObjNum 472 snap 110 seq_num 472) 2023-12-15T23:00:58.416 INFO:tasks.rados.rados.0.smithi071.stdout:1122: snap_remove snap 125 2023-12-15T23:00:58.724 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T23:00:58.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:58.826+0000 7f25c38f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:58.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:00:58.830+0000 7f25c38f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:00:59.350 INFO:tasks.rados.rados.0.smithi071.stdout:1122: done (2 left) 2023-12-15T23:00:59.350 INFO:tasks.rados.rados.0.smithi071.stdout:1123: snap_create 2023-12-15T23:00:59.350 INFO:tasks.rados.rados.0.smithi071.stdout:1119: finishing write tid 1 to smithi07183189-1 2023-12-15T23:00:59.350 INFO:tasks.rados.rados.0.smithi071.stdout:1119: finishing write tid 2 to smithi07183189-1 2023-12-15T23:00:59.350 INFO:tasks.rados.rados.0.smithi071.stdout:1121: expect (ObjNum 472 snap 110 seq_num 472) 2023-12-15T23:00:59.568 INFO:tasks.rados.rados.0.smithi071.stdout:1119: finishing write tid 3 to smithi07183189-1 2023-12-15T23:00:59.568 INFO:tasks.rados.rados.0.smithi071.stdout:1119: finishing write tid 4 to smithi07183189-1 2023-12-15T23:00:59.568 INFO:tasks.rados.rados.0.smithi071.stdout:1119: oid 1 updating version 0 to 103 2023-12-15T23:00:59.568 INFO:tasks.rados.rados.0.smithi071.stdout:1119: oid 1 updating version 103 to 104 2023-12-15T23:00:59.568 INFO:tasks.rados.rados.0.smithi071.stdout:1119: oid 1 updating version 104 to 105 2023-12-15T23:00:59.568 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 1 v 105 (ObjNum 543 snap 127 seq_num 543) dirty exists 2023-12-15T23:00:59.568 INFO:tasks.rados.rados.0.smithi071.stdout:1119: left oid 1 (ObjNum 543 snap 127 seq_num 543) 2023-12-15T23:01:00.351 INFO:tasks.rados.rados.0.smithi071.stdout:1119: done (2 left) 2023-12-15T23:01:00.352 INFO:tasks.rados.rados.0.smithi071.stdout:1121: done (1 left) 2023-12-15T23:01:00.352 INFO:tasks.rados.rados.0.smithi071.stdout:1123: done (0 left) 2023-12-15T23:01:00.352 INFO:tasks.rados.rados.0.smithi071.stdout:1124: write oid 48 current snap is 128 2023-12-15T23:01:00.352 INFO:tasks.rados.rados.0.smithi071.stdout:1124: seq_num 545 ranges {759815=471768,1742451=465507,2917178=242637} 2023-12-15T23:01:00.358 INFO:tasks.rados.rados.0.smithi071.stdout:1124: writing smithi07183189-48 from 759815 to 1231583 tid 1 2023-12-15T23:01:00.364 INFO:tasks.rados.rados.0.smithi071.stdout:1124: writing smithi07183189-48 from 1742451 to 2207958 tid 2 2023-12-15T23:01:00.369 INFO:tasks.rados.rados.0.smithi071.stdout:1124: writing smithi07183189-48 from 2917178 to 3159815 tid 3 2023-12-15T23:01:00.370 INFO:tasks.rados.rados.0.smithi071.stdout:1125: snap_remove snap 127 2023-12-15T23:01:00.900 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T23:01:01.005 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:01:01.002+0000 7f54bf02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:01:01.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:01:01.006+0000 7f54bf02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:01:01.372 INFO:tasks.rados.rados.0.smithi071.stdout:1125: done (1 left) 2023-12-15T23:01:01.372 INFO:tasks.rados.rados.0.smithi071.stdout:1126: read oid 12 snap 126 2023-12-15T23:01:01.372 INFO:tasks.rados.rados.0.smithi071.stdout:1126: expect (ObjNum 542 snap 126 seq_num 542) 2023-12-15T23:01:01.372 INFO:tasks.rados.rados.0.smithi071.stdout:1127: write oid 38 current snap is 128 2023-12-15T23:01:01.372 INFO:tasks.rados.rados.0.smithi071.stdout:1124: finishing write tid 1 to smithi07183189-48 2023-12-15T23:01:01.372 INFO:tasks.rados.rados.0.smithi071.stdout:1127: seq_num 546 ranges {776622=547017,1800933=725518,2944601=232021} 2023-12-15T23:01:01.379 INFO:tasks.rados.rados.0.smithi071.stdout:1127: writing smithi07183189-38 from 776622 to 1323639 tid 1 2023-12-15T23:01:01.386 INFO:tasks.rados.rados.0.smithi071.stdout:1127: writing smithi07183189-38 from 1800933 to 2526451 tid 2 2023-12-15T23:01:01.390 INFO:tasks.rados.rados.0.smithi071.stdout:1127: writing smithi07183189-38 from 2944601 to 3176622 tid 3 2023-12-15T23:01:01.390 INFO:tasks.rados.rados.0.smithi071.stdout:1128: delete oid 22 current snap is 128 2023-12-15T23:01:01.390 INFO:tasks.rados.rados.0.smithi071.stdout:1124: finishing write tid 2 to smithi07183189-48 2023-12-15T23:01:01.390 INFO:tasks.rados.rados.0.smithi071.stdout:1124: finishing write tid 3 to smithi07183189-48 2023-12-15T23:01:01.390 INFO:tasks.rados.rados.0.smithi071.stdout:1124: finishing write tid 4 to smithi07183189-48 2023-12-15T23:01:01.391 INFO:tasks.rados.rados.0.smithi071.stdout:1124: finishing write tid 5 to smithi07183189-48 2023-12-15T23:01:01.391 INFO:tasks.rados.rados.0.smithi071.stdout:1124: oid 48 updating version 0 to 259 2023-12-15T23:01:01.391 INFO:tasks.rados.rados.0.smithi071.stdout:1124: oid 48 version 259 is already newer than 258 2023-12-15T23:01:01.391 INFO:tasks.rados.rados.0.smithi071.stdout:1124: oid 48 version 259 is already newer than 257 2023-12-15T23:01:01.391 INFO:tasks.rados.rados.0.smithi071.stdout:1124: oid 48 version 259 is already newer than 257 2023-12-15T23:01:01.391 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 48 v 259 (ObjNum 545 snap 128 seq_num 545) dirty exists 2023-12-15T23:01:01.391 INFO:tasks.rados.rados.0.smithi071.stdout:1124: left oid 48 (ObjNum 545 snap 128 seq_num 545) 2023-12-15T23:01:01.479 INFO:tasks.rados.rados.0.smithi071.stdout:1126: expect (ObjNum 542 snap 126 seq_num 542) 2023-12-15T23:01:01.610 INFO:tasks.rados.rados.0.smithi071.stdout:1127: finishing write tid 1 to smithi07183189-38 2023-12-15T23:01:01.610 INFO:tasks.rados.rados.0.smithi071.stdout:1127: finishing write tid 2 to smithi07183189-38 2023-12-15T23:01:01.610 INFO:tasks.rados.rados.0.smithi071.stdout:1127: finishing write tid 3 to smithi07183189-38 2023-12-15T23:01:01.610 INFO:tasks.rados.rados.0.smithi071.stdout:1127: finishing write tid 4 to smithi07183189-38 2023-12-15T23:01:01.611 INFO:tasks.rados.rados.0.smithi071.stdout:1127: finishing write tid 5 to smithi07183189-38 2023-12-15T23:01:01.611 INFO:tasks.rados.rados.0.smithi071.stdout:1127: oid 38 updating version 0 to 170 2023-12-15T23:01:01.611 INFO:tasks.rados.rados.0.smithi071.stdout:1127: oid 38 updating version 170 to 172 2023-12-15T23:01:01.611 INFO:tasks.rados.rados.0.smithi071.stdout:1127: oid 38 version 172 is already newer than 172 2023-12-15T23:01:01.611 INFO:tasks.rados.rados.0.smithi071.stdout:1127: oid 38 updating version 172 to 173 2023-12-15T23:01:01.611 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 38 v 173 (ObjNum 546 snap 128 seq_num 546) dirty exists 2023-12-15T23:01:01.611 INFO:tasks.rados.rados.0.smithi071.stdout:1127: left oid 38 (ObjNum 546 snap 128 seq_num 546) 2023-12-15T23:01:01.611 INFO:tasks.rados.rados.0.smithi071.stdout:1124: done (3 left) 2023-12-15T23:01:01.611 INFO:tasks.rados.rados.0.smithi071.stdout:1126: done (2 left) 2023-12-15T23:01:01.611 INFO:tasks.rados.rados.0.smithi071.stdout:1127: done (1 left) 2023-12-15T23:01:01.611 INFO:tasks.rados.rados.0.smithi071.stdout:1128: done (0 left) 2023-12-15T23:01:01.611 INFO:tasks.rados.rados.0.smithi071.stdout:1129: snap_remove snap 126 2023-12-15T23:01:02.286 INFO:tasks.rados.rados.0.smithi071.stdout:1129: done (0 left) 2023-12-15T23:01:02.286 INFO:tasks.rados.rados.0.smithi071.stdout:1130: write (excl) oid 7 current snap is 128 2023-12-15T23:01:02.286 INFO:tasks.rados.rados.0.smithi071.stdout:1130: seq_num 548 ranges {410236=413868,1234250=729187,2715800=423063} 2023-12-15T23:01:02.289 INFO:tasks.rados.rados.0.smithi071.stdout:1130: writing smithi07183189-7 from 410236 to 824104 tid 1 2023-12-15T23:01:02.296 INFO:tasks.rados.rados.0.smithi071.stdout:1130: writing smithi07183189-7 from 1234250 to 1963437 tid 2 2023-12-15T23:01:02.303 INFO:tasks.rados.rados.0.smithi071.stdout:1130: writing smithi07183189-7 from 2715800 to 3138863 tid 3 2023-12-15T23:01:02.303 INFO:tasks.rados.rados.0.smithi071.stdout:1131: read oid 23 snap -1 2023-12-15T23:01:02.303 INFO:tasks.rados.rados.0.smithi071.stdout:1131: expect (ObjNum 522 snap 121 seq_num 522) 2023-12-15T23:01:02.303 INFO:tasks.rados.rados.0.smithi071.stdout:1132: delete oid 15 current snap is 128 2023-12-15T23:01:02.360 INFO:tasks.rados.rados.0.smithi071.stdout:1131: expect (ObjNum 522 snap 121 seq_num 522) 2023-12-15T23:01:02.498 INFO:tasks.rados.rados.0.smithi071.stdout:1130: finishing write tid 1 to smithi07183189-7 2023-12-15T23:01:02.498 INFO:tasks.rados.rados.0.smithi071.stdout:1130: finishing write tid 2 to smithi07183189-7 2023-12-15T23:01:02.498 INFO:tasks.rados.rados.0.smithi071.stdout:1130: finishing write tid 3 to smithi07183189-7 2023-12-15T23:01:02.498 INFO:tasks.rados.rados.0.smithi071.stdout:1130: finishing write tid 4 to smithi07183189-7 2023-12-15T23:01:02.498 INFO:tasks.rados.rados.0.smithi071.stdout:1130: finishing write tid 5 to smithi07183189-7 2023-12-15T23:01:02.499 INFO:tasks.rados.rados.0.smithi071.stdout:1130: oid 7 updating version 0 to 361 2023-12-15T23:01:02.499 INFO:tasks.rados.rados.0.smithi071.stdout:1130: oid 7 updating version 361 to 362 2023-12-15T23:01:02.499 INFO:tasks.rados.rados.0.smithi071.stdout:1130: oid 7 updating version 362 to 363 2023-12-15T23:01:02.499 INFO:tasks.rados.rados.0.smithi071.stdout:1130: oid 7 version 363 is already newer than 360 2023-12-15T23:01:02.499 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 7 v 363 (ObjNum 548 snap 128 seq_num 548) dirty exists 2023-12-15T23:01:02.499 INFO:tasks.rados.rados.0.smithi071.stdout:1130: left oid 7 (ObjNum 548 snap 128 seq_num 548) 2023-12-15T23:01:02.499 INFO:tasks.rados.rados.0.smithi071.stdout:1130: done (2 left) 2023-12-15T23:01:02.499 INFO:tasks.rados.rados.0.smithi071.stdout:1131: done (1 left) 2023-12-15T23:01:02.499 INFO:tasks.rados.rados.0.smithi071.stdout:1132: done (0 left) 2023-12-15T23:01:02.499 INFO:tasks.rados.rados.0.smithi071.stdout:1133: snap_create 2023-12-15T23:01:03.078 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T23:01:03.175 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T23:01:03.175 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T23:01:03.175 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T23:01:03.186 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:01:03.184+0000 7f3d5215e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:01:03.189 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:01:03.188+0000 7f3d5215e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:01:03.284 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:01:03.281+0000 7fcfe2f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:01:03.287 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:01:03.286+0000 7fcfe2f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:01:03.380 INFO:tasks.rados.rados.0.smithi071.stdout:1133: done (0 left) 2023-12-15T23:01:03.380 INFO:tasks.rados.rados.0.smithi071.stdout:1134: write oid 36 current snap is 129 2023-12-15T23:01:03.380 INFO:tasks.rados.rados.0.smithi071.stdout:1134: seq_num 550 ranges {443850=564366,1634861=732856,2770646=473204} 2023-12-15T23:01:03.386 INFO:tasks.rados.rados.0.smithi071.stdout:1134: writing smithi07183189-36 from 443850 to 1008216 tid 1 2023-12-15T23:01:03.394 INFO:tasks.rados.rados.0.smithi071.stdout:1134: writing smithi07183189-36 from 1634861 to 2367717 tid 2 2023-12-15T23:01:03.399 INFO:tasks.rados.rados.0.smithi071.stdout:1134: writing smithi07183189-36 from 2770646 to 3243850 tid 3 2023-12-15T23:01:03.399 INFO:tasks.rados.rados.0.smithi071.stdout:1135: write oid 5 current snap is 129 2023-12-15T23:01:03.399 INFO:tasks.rados.rados.0.smithi071.stdout:1135: seq_num 551 ranges {460657=639615,1693343=476514,2681716=578941} 2023-12-15T23:01:03.405 INFO:tasks.rados.rados.0.smithi071.stdout:1135: writing smithi07183189-5 from 460657 to 1100272 tid 1 2023-12-15T23:01:03.411 INFO:tasks.rados.rados.0.smithi071.stdout:1135: writing smithi07183189-5 from 1693343 to 2169857 tid 2 2023-12-15T23:01:03.417 INFO:tasks.rados.rados.0.smithi071.stdout:1135: writing smithi07183189-5 from 2681716 to 3260657 tid 3 2023-12-15T23:01:03.417 INFO:tasks.rados.rados.0.smithi071.stdout:1136: snap_create 2023-12-15T23:01:03.467 INFO:tasks.rados.rados.0.smithi071.stdout:1134: finishing write tid 1 to smithi07183189-36 2023-12-15T23:01:03.501 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-12-15T23:01:03.501 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-12-15T23:01:03.504 INFO:tasks.rados.rados.0.smithi071.stdout:1134: finishing write tid 2 to smithi07183189-36 2023-12-15T23:01:03.513 INFO:tasks.rados.rados.0.smithi071.stdout:1135: finishing write tid 1 to smithi07183189-5 2023-12-15T23:01:03.514 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":405,"stamp":"2023-12-15T23:01:03.376122+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":89149289,"num_objects":363,"num_object_clones":265,"num_object_copies":1089,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":65,"num_read":2155,"num_read_kb":1536064,"num_write":1826,"num_write_kb":494147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3612,"ondisk_log_size":3612,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":378904,"kb_used_data":238328,"kb_used_omap":0,"kb_used_meta":140544,"kb_avail":377108456,"statfs":{"total":386547056640,"available":386159058944,"internally_reserved":0,"allocated":244047872,"data_stored":257419751,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":143917056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":420175,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":15,"num_read_kb":11747,"num_write":8,"num_write_kb":2603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.027589"},"pg_stats":[{"pgid":"3.f","version":"271'107","reported_seq":654,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-12-15T23:01:00.371132+0000","last_change":"2023-12-15T23:00:59.388770+0000","last_active":"2023-12-15T23:01:00.371132+0000","last_peered":"2023-12-15T23:01:00.371132+0000","last_clean":"2023-12-15T23:01:00.371132+0000","last_became_active":"2023-12-15T22:56:39.503797+0000","last_became_peered":"2023-12-15T22:56:39.503797+0000","last_unstale":"2023-12-15T23:01:00.371132+0000","last_undegraded":"2023-12-15T23:01:00.371132+0000","last_fullsized":"2023-12-15T23:01:00.371132+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6369985,"num_objects":13,"num_object_clones":9,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":105,"num_read_kb":90405,"num_write":48,"num_write_kb":14372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":516,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-12-15T23:00:58.551354+0000","last_change":"2023-12-15T23:00:57.363695+0000","last_active":"2023-12-15T23:00:58.551354+0000","last_peered":"2023-12-15T23:00:58.551354+0000","last_clean":"2023-12-15T23:00:58.551354+0000","last_became_active":"2023-12-15T22:56:39.552856+0000","last_became_peered":"2023-12-15T22:56:39.552856+0000","last_unstale":"2023-12-15T23:00:58.551354+0000","last_undegraded":"2023-12-15T23:00:58.551354+0000","last_fullsized":"2023-12-15T23:00:58.551354+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"266'358","reported_seq":837,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-12-15T23:00:58.362889+0000","last_change":"2023-12-15T23:00:57.641356+0000","last_active":"2023-12-15T23:00:58.362889+0000","last_peered":"2023-12-15T23:00:58.362889+0000","last_clean":"2023-12-15T23:00:58.362889+0000","last_became_active":"2023-12-15T22:56:39.495490+0000","last_became_peered":"2023-12-15T22:56:39.495490+0000","last_unstale":"2023-12-15T23:00:58.362889+0000","last_undegraded":"2023-12-15T23:00:58.362889+0000","last_fullsized":"2023-12-15T23:00:58.362889+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13513323,"num_objects":39,"num_object_clones":27,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":273,"num_read_kb":230324,"num_write":199,"num_write_kb":57479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"264'325","reported_seq":785,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-12-15T23:01:00.375916+0000","last_change":"2023-12-15T23:00:59.365761+0000","last_active":"2023-12-15T23:01:00.375916+0000","last_peered":"2023-12-15T23:01:00.375916+0000","last_clean":"2023-12-15T23:01:00.375916+0000","last_became_active":"2023-12-15T22:56:39.579234+0000","last_became_peered":"2023-12-15T22:56:39.579234+0000","last_unstale":"2023-12-15T23:01:00.375916+0000","last_undegraded":"2023-12-15T23:01:00.375916+0000","last_fullsized":"2023-12-15T23:01:00.375916+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2907710,"num_objects":31,"num_object_clones":28,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":162,"num_read_kb":112883,"num_write":167,"num_write_kb":48480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"268'186","reported_seq":663,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-12-15T23:00:58.373557+0000","last_change":"2023-12-15T23:00:57.702594+0000","last_active":"2023-12-15T23:00:58.373557+0000","last_peered":"2023-12-15T23:00:58.373557+0000","last_clean":"2023-12-15T23:00:58.373557+0000","last_became_active":"2023-12-15T22:56:39.572176+0000","last_became_peered":"2023-12-15T22:56:39.572176+0000","last_unstale":"2023-12-15T23:00:58.373557+0000","last_undegraded":"2023-12-15T23:00:58.373557+0000","last_fullsized":"2023-12-15T23:00:58.373557+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6050718,"num_objects":19,"num_object_clones":12,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":78,"num_read_kb":45678,"num_write":90,"num_write_kb":26466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"266'173","reported_seq":661,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-12-15T23:01:00.362985+0000","last_change":"2023-12-15T23:00:59.365879+0000","last_active":"2023-12-15T23:01:00.362985+0000","last_peered":"2023-12-15T23:01:00.362985+0000","last_clean":"2023-12-15T23:01:00.362985+0000","last_became_active":"2023-12-15T22:56:39.516361+0000","last_became_peered":"2023-12-15T22:56:39.516361+0000","last_unstale":"2023-12-15T23:01:00.362985+0000","last_undegraded":"2023-12-15T23:01:00.362985+0000","last_fullsized":"2023-12-15T23:01:00.362985+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5781806,"num_objects":17,"num_object_clones":12,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":74,"num_read_kb":56864,"num_write":73,"num_write_kb":21081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"266'651","reported_seq":948,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-12-15T23:01:00.361413+0000","last_change":"2023-12-15T23:00:59.365596+0000","last_active":"2023-12-15T23:01:00.361413+0000","last_peered":"2023-12-15T23:01:00.361413+0000","last_clean":"2023-12-15T23:01:00.361413+0000","last_became_active":"2023-12-15T22:56:39.519748+0000","last_became_peered":"2023-12-15T22:56:39.519748+0000","last_unstale":"2023-12-15T23:01:00.361413+0000","last_undegraded":"2023-12-15T23:01:00.361413+0000","last_fullsized":"2023-12-15T23:01:00.361413+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":651,"log_dups_size":0,"ondisk_log_size":651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14034307,"num_objects":66,"num_object_clones":51,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":284,"num_read_kb":177635,"num_write":279,"num_write_kb":79789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"266'168","reported_seq":689,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-12-15T23:00:58.353530+0000","last_change":"2023-12-15T23:00:57.638764+0000","last_active":"2023-12-15T23:00:58.353530+0000","last_peered":"2023-12-15T23:00:58.353530+0000","last_clean":"2023-12-15T23:00:58.353530+0000","last_became_active":"2023-12-15T22:56:39.543179+0000","last_became_peered":"2023-12-15T22:56:39.543179+0000","last_unstale":"2023-12-15T23:00:58.353530+0000","last_undegraded":"2023-12-15T23:00:58.353530+0000","last_fullsized":"2023-12-15T23:00:58.353530+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8101238,"num_objects":19,"num_object_clones":13,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":162,"num_read_kb":140548,"num_write":96,"num_write_kb":26202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":550,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-12-15T23:00:58.372274+0000","last_change":"2023-12-15T22:56:32.607017+0000","last_active":"2023-12-15T23:00:58.372274+0000","last_peered":"2023-12-15T23:00:58.372274+0000","last_clean":"2023-12-15T23:00:58.372274+0000","last_became_active":"2023-12-15T22:56:32.602384+0000","last_became_peered":"2023-12-15T22:56:32.602384+0000","last_unstale":"2023-12-15T23:00:58.372274+0000","last_undegraded":"2023-12-15T23:00:58.372274+0000","last_fullsized":"2023-12-15T23:00:58.372274+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:31.525572+0000","last_clean_scrub_stamp":"2023-12-15T22:56:31.525572+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":409,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-12-15T23:00:58.365573+0000","last_change":"2023-12-15T22:56:33.627481+0000","last_active":"2023-12-15T23:00:58.365573+0000","last_peered":"2023-12-15T23:00:58.365573+0000","last_clean":"2023-12-15T23:00:58.365573+0000","last_became_active":"2023-12-15T22:56:33.624689+0000","last_became_peered":"2023-12-15T22:56:33.624689+0000","last_unstale":"2023-12-15T23:00:58.365573+0000","last_undegraded":"2023-12-15T23:00:58.365573+0000","last_fullsized":"2023-12-15T23:00:58.365573+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"266'254","reported_seq":766,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-12-15T23:00:58.365097+0000","last_change":"2023-12-15T23:00:57.638254+0000","last_active":"2023-12-15T23:00:58.365097+0000","last_peered":"2023-12-15T23:00:58.365097+0000","last_clean":"2023-12-15T23:00:58.365097+0000","last_became_active":"2023-12-15T22:56:39.529359+0000","last_became_peered":"2023-12-15T22:56:39.529359+0000","last_unstale":"2023-12-15T23:00:58.365097+0000","last_undegraded":"2023-12-15T23:00:58.365097+0000","last_fullsized":"2023-12-15T23:00:58.365097+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":254,"log_dups_size":0,"ondisk_log_size":254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5311210,"num_objects":31,"num_object_clones":20,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":207,"num_read_kb":148055,"num_write":153,"num_write_kb":44413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":443,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-12-15T23:01:00.368112+0000","last_change":"2023-12-15T22:56:33.655797+0000","last_active":"2023-12-15T23:01:00.368112+0000","last_peered":"2023-12-15T23:01:00.368112+0000","last_clean":"2023-12-15T23:01:00.368112+0000","last_became_active":"2023-12-15T22:56:33.652885+0000","last_became_peered":"2023-12-15T22:56:33.652885+0000","last_unstale":"2023-12-15T23:01:00.368112+0000","last_undegraded":"2023-12-15T23:01:00.368112+0000","last_fullsized":"2023-12-15T23:01:00.368112+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"267'318","reported_seq":755,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-12-15T23:00:58.376440+0000","last_change":"2023-12-15T23:00:57.709262+0000","last_active":"2023-12-15T23:00:58.376440+0000","last_peered":"2023-12-15T23:00:58.376440+0000","last_clean":"2023-12-15T23:00:58.376440+0000","last_became_active":"2023-12-15T22:56:39.543509+0000","last_became_peered":"2023-12-15T22:56:39.543509+0000","last_unstale":"2023-12-15T23:00:58.376440+0000","last_undegraded":"2023-12-15T23:00:58.376440+0000","last_fullsized":"2023-12-15T23:00:58.376440+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":318,"log_dups_size":0,"ondisk_log_size":318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7613868,"num_objects":35,"num_object_clones":24,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":182,"num_read_kb":134406,"num_write":140,"num_write_kb":42979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":409,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-12-15T23:00:58.365357+0000","last_change":"2023-12-15T22:56:33.600485+0000","last_active":"2023-12-15T23:00:58.365357+0000","last_peered":"2023-12-15T23:00:58.365357+0000","last_clean":"2023-12-15T23:00:58.365357+0000","last_became_active":"2023-12-15T22:56:33.597320+0000","last_became_peered":"2023-12-15T22:56:33.597320+0000","last_unstale":"2023-12-15T23:00:58.365357+0000","last_undegraded":"2023-12-15T23:00:58.365357+0000","last_fullsized":"2023-12-15T23:00:58.365357+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"267'157","reported_seq":646,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-12-15T23:00:58.375197+0000","last_change":"2023-12-15T23:00:57.704460+0000","last_active":"2023-12-15T23:00:58.375197+0000","last_peered":"2023-12-15T23:00:58.375197+0000","last_clean":"2023-12-15T23:00:58.375197+0000","last_became_active":"2023-12-15T22:56:39.560980+0000","last_became_peered":"2023-12-15T22:56:39.560980+0000","last_unstale":"2023-12-15T23:00:58.375197+0000","last_undegraded":"2023-12-15T23:00:58.375197+0000","last_fullsized":"2023-12-15T23:00:58.375197+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3075780,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":84,"num_read_kb":65658,"num_write":75,"num_write_kb":21880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":443,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-12-15T23:01:00.364147+0000","last_change":"2023-12-15T22:56:33.648541+0000","last_active":"2023-12-15T23:01:00.364147+0000","last_peered":"2023-12-15T23:01:00.364147+0000","last_clean":"2023-12-15T23:01:00.364147+0000","last_became_active":"2023-12-15T22:56:33.643681+0000","last_became_peered":"2023-12-15T22:56:33.643681+0000","last_unstale":"2023-12-15T23:01:00.364147+0000","last_undegraded":"2023-12-15T23:01:00.364147+0000","last_fullsized":"2023-12-15T23:01:00.364147+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"245'241","reported_seq":685,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-12-15T23:00:58.546274+0000","last_change":"2023-12-15T23:00:57.361308+0000","last_active":"2023-12-15T23:00:58.546274+0000","last_peered":"2023-12-15T23:00:58.546274+0000","last_clean":"2023-12-15T23:00:58.546274+0000","last_became_active":"2023-12-15T22:56:39.506510+0000","last_became_peered":"2023-12-15T22:56:39.506510+0000","last_unstale":"2023-12-15T23:00:58.546274+0000","last_undegraded":"2023-12-15T23:00:58.546274+0000","last_fullsized":"2023-12-15T23:00:58.546274+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5496054,"num_objects":26,"num_object_clones":18,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":150,"num_read_kb":108658,"num_write":107,"num_write_kb":31471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"242'110","reported_seq":587,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-12-15T23:00:58.435687+0000","last_change":"2023-12-15T23:00:57.641105+0000","last_active":"2023-12-15T23:00:58.435687+0000","last_peered":"2023-12-15T23:00:58.435687+0000","last_clean":"2023-12-15T23:00:58.435687+0000","last_became_active":"2023-12-15T22:56:39.511159+0000","last_became_peered":"2023-12-15T22:56:39.511159+0000","last_unstale":"2023-12-15T23:00:58.435687+0000","last_undegraded":"2023-12-15T23:00:58.435687+0000","last_fullsized":"2023-12-15T23:00:58.435687+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3932904,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":42,"num_read_kb":33126,"num_write":47,"num_write_kb":14375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":397,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-12-15T23:00:58.549595+0000","last_change":"2023-12-15T22:56:33.633890+0000","last_active":"2023-12-15T23:00:58.549595+0000","last_peered":"2023-12-15T23:00:58.549595+0000","last_clean":"2023-12-15T23:00:58.549595+0000","last_became_active":"2023-12-15T22:56:33.631134+0000","last_became_peered":"2023-12-15T22:56:33.631134+0000","last_unstale":"2023-12-15T23:00:58.549595+0000","last_undegraded":"2023-12-15T23:00:58.549595+0000","last_fullsized":"2023-12-15T23:00:58.549595+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"245'173","reported_seq":631,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-12-15T23:00:58.547457+0000","last_change":"2023-12-15T23:00:57.364540+0000","last_active":"2023-12-15T23:00:58.547457+0000","last_peered":"2023-12-15T23:00:58.547457+0000","last_clean":"2023-12-15T23:00:58.547457+0000","last_became_active":"2023-12-15T22:56:39.532730+0000","last_became_peered":"2023-12-15T22:56:39.532730+0000","last_unstale":"2023-12-15T23:00:58.547457+0000","last_undegraded":"2023-12-15T23:00:58.547457+0000","last_fullsized":"2023-12-15T23:00:58.547457+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":14,"num_object_clones":12,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":87,"num_read_kb":65976,"num_write":82,"num_write_kb":25453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":421,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-12-15T23:00:58.376607+0000","last_change":"2023-12-15T22:56:33.655736+0000","last_active":"2023-12-15T23:00:58.376607+0000","last_peered":"2023-12-15T23:00:58.376607+0000","last_clean":"2023-12-15T23:00:58.376607+0000","last_became_active":"2023-12-15T22:56:33.652917+0000","last_became_peered":"2023-12-15T22:56:33.652917+0000","last_unstale":"2023-12-15T23:00:58.376607+0000","last_undegraded":"2023-12-15T23:00:58.376607+0000","last_fullsized":"2023-12-15T23:00:58.376607+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"267'211","reported_seq":694,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-12-15T23:00:58.374996+0000","last_change":"2023-12-15T23:00:57.704812+0000","last_active":"2023-12-15T23:00:58.374996+0000","last_peered":"2023-12-15T23:00:58.374996+0000","last_clean":"2023-12-15T23:00:58.374996+0000","last_became_active":"2023-12-15T22:56:39.557252+0000","last_became_peered":"2023-12-15T22:56:39.557252+0000","last_unstale":"2023-12-15T23:00:58.374996+0000","last_undegraded":"2023-12-15T23:00:58.374996+0000","last_fullsized":"2023-12-15T23:00:58.374996+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":211,"log_dups_size":0,"ondisk_log_size":211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6370018,"num_objects":19,"num_object_clones":13,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":127,"num_read_kb":96363,"num_write":98,"num_write_kb":27302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-12-15T23:00:58.548617+0000","last_change":"2023-12-15T22:56:33.632348+0000","last_active":"2023-12-15T23:00:58.548617+0000","last_peered":"2023-12-15T23:00:58.548617+0000","last_clean":"2023-12-15T23:00:58.548617+0000","last_became_active":"2023-12-15T22:56:33.629022+0000","last_became_peered":"2023-12-15T22:56:33.629022+0000","last_unstale":"2023-12-15T23:00:58.548617+0000","last_undegraded":"2023-12-15T23:00:58.548617+0000","last_fullsized":"2023-12-15T23:00:58.548617+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"245'83","reported_seq":565,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-12-15T23:00:58.548277+0000","last_change":"2023-12-15T23:00:57.363851+0000","last_active":"2023-12-15T23:00:58.548277+0000","last_peered":"2023-12-15T23:00:58.548277+0000","last_clean":"2023-12-15T23:00:58.548277+0000","last_became_active":"2023-12-15T22:56:39.535145+0000","last_became_peered":"2023-12-15T22:56:39.535145+0000","last_unstale":"2023-12-15T23:00:58.548277+0000","last_undegraded":"2023-12-15T23:00:58.548277+0000","last_fullsized":"2023-12-15T23:00:58.548277+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:38.356579+0000","last_clean_scrub_stamp":"2023-12-15T22:56:38.356579+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":6,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":29394,"num_write":35,"num_write_kb":10511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":409,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-12-15T23:00:58.355212+0000","last_change":"2023-12-15T22:56:33.627005+0000","last_active":"2023-12-15T23:00:58.355212+0000","last_peered":"2023-12-15T23:00:58.355212+0000","last_clean":"2023-12-15T23:00:58.355212+0000","last_became_active":"2023-12-15T22:56:33.622181+0000","last_became_peered":"2023-12-15T22:56:33.622181+0000","last_unstale":"2023-12-15T23:00:58.355212+0000","last_undegraded":"2023-12-15T23:00:58.355212+0000","last_fullsized":"2023-12-15T23:00:58.355212+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-15T22:56:32.529733+0000","last_clean_scrub_stamp":"2023-12-15T22:56:32.529733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":88558921,"num_objects":361,"num_object_clones":265,"num_object_copies":1083,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":65,"num_read":2052,"num_read_kb":1535973,"num_write":1689,"num_write_kb":492253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3515,"ondisk_log_size":3515,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705934,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":97060,"kb_used_data":61916,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94274780,"statfs":{"total":96636764160,"available":96537374720,"internally_reserved":0,"allocated":63401984,"data_stored":66633770,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705933,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":78364,"kb_used_data":43220,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94293476,"statfs":{"total":96636764160,"available":96556519424,"internally_reserved":0,"allocated":44257280,"data_stored":46848638,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705933,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":104316,"kb_used_data":69172,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94267524,"statfs":{"total":96636764160,"available":96529944576,"internally_reserved":0,"allocated":70832128,"data_stored":74914276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705934,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":99164,"kb_used_data":64020,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94272676,"statfs":{"total":96636764160,"available":96535220224,"internally_reserved":0,"allocated":65556480,"data_stored":69023067,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-15T23:01:03.537 INFO:tasks.rados.rados.0.smithi071.stdout:1134: finishing write tid 3 to smithi07183189-36 2023-12-15T23:01:03.539 INFO:tasks.rados.rados.0.smithi071.stdout:1135: finishing write tid 2 to smithi07183189-5 2023-12-15T23:01:03.543 INFO:tasks.rados.rados.0.smithi071.stdout:1134: finishing write tid 4 to smithi07183189-36 2023-12-15T23:01:03.544 INFO:tasks.rados.rados.0.smithi071.stdout:1134: finishing write tid 5 to smithi07183189-36 2023-12-15T23:01:03.544 INFO:tasks.rados.rados.0.smithi071.stdout:1134: oid 36 updating version 0 to 368 2023-12-15T23:01:03.545 INFO:tasks.rados.rados.0.smithi071.stdout:1134: oid 36 version 368 is already newer than 368 2023-12-15T23:01:03.545 INFO:tasks.rados.rados.0.smithi071.stdout:1134: oid 36 version 368 is already newer than 365 2023-12-15T23:01:03.545 INFO:tasks.rados.rados.0.smithi071.stdout:1134: oid 36 version 368 is already newer than 366 2023-12-15T23:01:03.545 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 36 v 368 (ObjNum 550 snap 129 seq_num 550) dirty exists 2023-12-15T23:01:03.545 INFO:tasks.rados.rados.0.smithi071.stdout:1134: left oid 36 (ObjNum 550 snap 129 seq_num 550) 2023-12-15T23:01:03.547 INFO:tasks.rados.rados.0.smithi071.stdout:1135: finishing write tid 3 to smithi07183189-5 2023-12-15T23:01:03.557 INFO:tasks.rados.rados.0.smithi071.stdout:1135: finishing write tid 4 to smithi07183189-5 2023-12-15T23:01:03.558 INFO:tasks.rados.rados.0.smithi071.stdout:1135: finishing write tid 5 to smithi07183189-5 2023-12-15T23:01:03.558 INFO:tasks.rados.rados.0.smithi071.stdout:1135: oid 5 updating version 0 to 652 2023-12-15T23:01:03.558 INFO:tasks.rados.rados.0.smithi071.stdout:1135: oid 5 updating version 652 to 654 2023-12-15T23:01:03.558 INFO:tasks.rados.rados.0.smithi071.stdout:1135: oid 5 updating version 654 to 655 2023-12-15T23:01:03.558 INFO:tasks.rados.rados.0.smithi071.stdout:1135: oid 5 version 655 is already newer than 654 2023-12-15T23:01:03.558 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 5 v 655 (ObjNum 551 snap 129 seq_num 551) dirty exists 2023-12-15T23:01:03.558 INFO:tasks.rados.rados.0.smithi071.stdout:1135: left oid 5 (ObjNum 551 snap 129 seq_num 551) 2023-12-15T23:01:04.386 INFO:tasks.rados.rados.0.smithi071.stdout:1134: done (2 left) 2023-12-15T23:01:04.386 INFO:tasks.rados.rados.0.smithi071.stdout:1135: done (1 left) 2023-12-15T23:01:04.386 INFO:tasks.rados.rados.0.smithi071.stdout:1136: done (0 left) 2023-12-15T23:01:04.386 INFO:tasks.rados.rados.0.smithi071.stdout:1137: write oid 34 current snap is 130 2023-12-15T23:01:04.386 INFO:tasks.rados.rados.0.smithi071.stdout:1137: seq_num 552 ranges {477464=714864,1751825=736525,3225492=51972} 2023-12-15T23:01:04.392 INFO:tasks.rados.rados.0.smithi071.stdout:1137: writing smithi07183189-34 from 477464 to 1192328 tid 1 2023-12-15T23:01:04.399 INFO:tasks.rados.rados.0.smithi071.stdout:1137: writing smithi07183189-34 from 1751825 to 2488350 tid 2 2023-12-15T23:01:04.404 INFO:tasks.rados.rados.0.smithi071.stdout:1137: writing smithi07183189-34 from 3225492 to 3277464 tid 3 2023-12-15T23:01:04.404 INFO:tasks.rados.rados.0.smithi071.stdout:1138: snap_create 2023-12-15T23:01:04.435 INFO:tasks.rados.rados.0.smithi071.stdout:1137: finishing write tid 1 to smithi07183189-34 2023-12-15T23:01:04.456 INFO:tasks.rados.rados.0.smithi071.stdout:1137: finishing write tid 2 to smithi07183189-34 2023-12-15T23:01:04.466 INFO:tasks.rados.rados.0.smithi071.stdout:1137: finishing write tid 3 to smithi07183189-34 2023-12-15T23:01:04.474 INFO:tasks.rados.rados.0.smithi071.stdout:1137: finishing write tid 4 to smithi07183189-34 2023-12-15T23:01:04.475 INFO:tasks.rados.rados.0.smithi071.stdout:1137: finishing write tid 5 to smithi07183189-34 2023-12-15T23:01:04.475 INFO:tasks.rados.rados.0.smithi071.stdout:1137: oid 34 updating version 0 to 86 2023-12-15T23:01:04.475 INFO:tasks.rados.rados.0.smithi071.stdout:1137: oid 34 version 86 is already newer than 84 2023-12-15T23:01:04.475 INFO:tasks.rados.rados.0.smithi071.stdout:1137: oid 34 updating version 86 to 87 2023-12-15T23:01:04.475 INFO:tasks.rados.rados.0.smithi071.stdout:1137: oid 34 version 87 is already newer than 85 2023-12-15T23:01:04.475 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 34 v 87 (ObjNum 552 snap 130 seq_num 552) dirty exists 2023-12-15T23:01:04.475 INFO:tasks.rados.rados.0.smithi071.stdout:1137: left oid 34 (ObjNum 552 snap 130 seq_num 552) 2023-12-15T23:01:05.261 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T23:01:05.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:01:05.376+0000 7f0555f04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:01:05.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:01:05.380+0000 7f0555f04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:01:05.395 INFO:tasks.rados.rados.0.smithi071.stdout:1137: done (1 left) 2023-12-15T23:01:05.395 INFO:tasks.rados.rados.0.smithi071.stdout:1138: done (0 left) 2023-12-15T23:01:05.395 INFO:tasks.rados.rados.0.smithi071.stdout:1139: snap_remove snap 130 2023-12-15T23:01:06.394 INFO:tasks.rados.rados.0.smithi071.stdout:1139: done (0 left) 2023-12-15T23:01:06.395 INFO:tasks.rados.rados.0.smithi071.stdout:1140: read oid 44 snap -1 2023-12-15T23:01:06.395 INFO:tasks.rados.rados.0.smithi071.stdout:1140: expect deleted 2023-12-15T23:01:06.395 INFO:tasks.rados.rados.0.smithi071.stdout:1141: write oid 12 current snap is 131 2023-12-15T23:01:06.395 INFO:tasks.rados.rados.0.smithi071.stdout:1141: seq_num 553 ranges {494271=790113,1693954=480183,2620209=512129} 2023-12-15T23:01:06.401 INFO:tasks.rados.rados.0.smithi071.stdout:1141: writing smithi07183189-12 from 494271 to 1284384 tid 1 2023-12-15T23:01:06.406 INFO:tasks.rados.rados.0.smithi071.stdout:1141: writing smithi07183189-12 from 1693954 to 2174137 tid 2 2023-12-15T23:01:06.412 INFO:tasks.rados.rados.0.smithi071.stdout:1141: writing smithi07183189-12 from 2620209 to 3132338 tid 3 2023-12-15T23:01:06.412 INFO:tasks.rados.rados.0.smithi071.stdout:1142: read oid 25 snap -1 2023-12-15T23:01:06.412 INFO:tasks.rados.rados.0.smithi071.stdout:1142: expect (ObjNum 502 snap 114 seq_num 502) 2023-12-15T23:01:06.412 INFO:tasks.rados.rados.0.smithi071.stdout:1143: read oid 36 snap -1 2023-12-15T23:01:06.412 INFO:tasks.rados.rados.0.smithi071.stdout:1143: expect (ObjNum 550 snap 129 seq_num 550) 2023-12-15T23:01:06.412 INFO:tasks.rados.rados.0.smithi071.stdout:1144: write oid 9 current snap is 131 2023-12-15T23:01:06.412 INFO:tasks.rados.rados.0.smithi071.stdout:1144: seq_num 554 ranges {511078=465362,1752436=740194,3163985=147093} 2023-12-15T23:01:06.418 INFO:tasks.rados.rados.0.smithi071.stdout:1144: writing smithi07183189-9 from 511078 to 976440 tid 1 2023-12-15T23:01:06.426 INFO:tasks.rados.rados.0.smithi071.stdout:1144: writing smithi07183189-9 from 1752436 to 2492630 tid 2 2023-12-15T23:01:06.431 INFO:tasks.rados.rados.0.smithi071.stdout:1144: writing smithi07183189-9 from 3163985 to 3311078 tid 3 2023-12-15T23:01:06.431 INFO:tasks.rados.rados.0.smithi071.stdout:1145: read oid 22 snap 123 2023-12-15T23:01:06.431 INFO:tasks.rados.rados.0.smithi071.stdout:1145: expect (ObjNum 429 snap 100 seq_num 429) 2023-12-15T23:01:06.431 INFO:tasks.rados.rados.0.smithi071.stdout:1146: write (excl) oid 15 current snap is 131 2023-12-15T23:01:06.431 INFO:tasks.rados.rados.0.smithi071.stdout:1146: seq_num 555 ranges {527885=656964,1927271=483852,3191408=136477} 2023-12-15T23:01:06.438 INFO:tasks.rados.rados.0.smithi071.stdout:1146: writing smithi07183189-15 from 527885 to 1184849 tid 1 2023-12-15T23:01:06.445 INFO:tasks.rados.rados.0.smithi071.stdout:1146: writing smithi07183189-15 from 1927271 to 2411123 tid 2 2023-12-15T23:01:06.451 INFO:tasks.rados.rados.0.smithi071.stdout:1146: writing smithi07183189-15 from 3191408 to 3327885 tid 3 2023-12-15T23:01:06.451 INFO:tasks.rados.rados.0.smithi071.stdout:1147: read oid 48 snap -1 2023-12-15T23:01:06.451 INFO:tasks.rados.rados.0.smithi071.stdout:1147: expect (ObjNum 545 snap 128 seq_num 545) 2023-12-15T23:01:06.451 INFO:tasks.rados.rados.0.smithi071.stdout:1148: write oid 35 current snap is 131 2023-12-15T23:01:06.451 INFO:tasks.rados.rados.0.smithi071.stdout:1148: seq_num 556 ranges {544692=732213,1985753=743863,3335184=9508} 2023-12-15T23:01:06.458 INFO:tasks.rados.rados.0.smithi071.stdout:1148: writing smithi07183189-35 from 544692 to 1276905 tid 1 2023-12-15T23:01:06.467 INFO:tasks.rados.rados.0.smithi071.stdout:1148: writing smithi07183189-35 from 1985753 to 2729616 tid 2 2023-12-15T23:01:06.471 INFO:tasks.rados.rados.0.smithi071.stdout:1148: writing smithi07183189-35 from 3335184 to 3344692 tid 3 2023-12-15T23:01:06.471 INFO:tasks.rados.rados.0.smithi071.stdout:1149: write oid 20 current snap is 131 2023-12-15T23:01:06.471 INFO:tasks.rados.rados.0.smithi071.stdout:1149: seq_num 557 ranges {561499=407462,1527882=487521,2729901=557217} 2023-12-15T23:01:06.476 INFO:tasks.rados.rados.0.smithi071.stdout:1149: writing smithi07183189-20 from 561499 to 968961 tid 1 2023-12-15T23:01:06.482 INFO:tasks.rados.rados.0.smithi071.stdout:1149: writing smithi07183189-20 from 1527882 to 2015403 tid 2 2023-12-15T23:01:06.490 INFO:tasks.rados.rados.0.smithi071.stdout:1149: writing smithi07183189-20 from 2729901 to 3287118 tid 3 2023-12-15T23:01:06.490 INFO:tasks.rados.rados.0.smithi071.stdout:1150: delete oid 13 current snap is 131 2023-12-15T23:01:06.490 INFO:tasks.rados.rados.0.smithi071.stdout:1145: expect (ObjNum 429 snap 100 seq_num 429) 2023-12-15T23:01:06.665 INFO:tasks.rados.rados.0.smithi071.stdout:1142: expect (ObjNum 502 snap 114 seq_num 502) 2023-12-15T23:01:06.826 INFO:tasks.rados.rados.0.smithi071.stdout:1144: finishing write tid 1 to smithi07183189-9 2023-12-15T23:01:06.826 INFO:tasks.rados.rados.0.smithi071.stdout:1141: finishing write tid 1 to smithi07183189-12 2023-12-15T23:01:06.826 INFO:tasks.rados.rados.0.smithi071.stdout:1146: finishing write tid 1 to smithi07183189-15 2023-12-15T23:01:06.826 INFO:tasks.rados.rados.0.smithi071.stdout:1143: expect (ObjNum 550 snap 129 seq_num 550) 2023-12-15T23:01:06.988 INFO:tasks.rados.rados.0.smithi071.stdout:1146: finishing write tid 2 to smithi07183189-15 2023-12-15T23:01:06.988 INFO:tasks.rados.rados.0.smithi071.stdout:1141: finishing write tid 2 to smithi07183189-12 2023-12-15T23:01:06.988 INFO:tasks.rados.rados.0.smithi071.stdout:1144: finishing write tid 2 to smithi07183189-9 2023-12-15T23:01:06.988 INFO:tasks.rados.rados.0.smithi071.stdout:1146: finishing write tid 3 to smithi07183189-15 2023-12-15T23:01:06.988 INFO:tasks.rados.rados.0.smithi071.stdout:1140: done (10 left) 2023-12-15T23:01:06.988 INFO:tasks.rados.rados.0.smithi071.stdout:1142: done (9 left) 2023-12-15T23:01:06.988 INFO:tasks.rados.rados.0.smithi071.stdout:1143: done (8 left) 2023-12-15T23:01:06.989 INFO:tasks.rados.rados.0.smithi071.stdout:1145: done (7 left) 2023-12-15T23:01:06.989 INFO:tasks.rados.rados.0.smithi071.stdout:1150: done (6 left) 2023-12-15T23:01:06.989 INFO:tasks.rados.rados.0.smithi071.stdout:1151: snap_remove snap 128 2023-12-15T23:01:07.404 INFO:tasks.rados.rados.0.smithi071.stdout:1151: done (6 left) 2023-12-15T23:01:07.404 INFO:tasks.rados.rados.0.smithi071.stdout:1152: read oid 13 snap 123 2023-12-15T23:01:07.404 INFO:tasks.rados.rados.0.smithi071.stdout:1152: expect deleted 2023-12-15T23:01:07.404 INFO:tasks.rados.rados.0.smithi071.stdout:1153: write oid 38 current snap is 131 2023-12-15T23:01:07.404 INFO:tasks.rados.rados.0.smithi071.stdout:1146: finishing write tid 4 to smithi07183189-15 2023-12-15T23:01:07.404 INFO:tasks.rados.rados.0.smithi071.stdout:1153: seq_num 559 ranges {595113=557960,1644846=491190,2784747=610366} 2023-12-15T23:01:07.410 INFO:tasks.rados.rados.0.smithi071.stdout:1153: writing smithi07183189-38 from 595113 to 1153073 tid 1 2023-12-15T23:01:07.416 INFO:tasks.rados.rados.0.smithi071.stdout:1153: writing smithi07183189-38 from 1644846 to 2136036 tid 2 2023-12-15T23:01:07.424 INFO:tasks.rados.rados.0.smithi071.stdout:1153: writing smithi07183189-38 from 2784747 to 3395113 tid 3 2023-12-15T23:01:07.424 INFO:tasks.rados.rados.0.smithi071.stdout:1154: read oid 17 snap -1 2023-12-15T23:01:07.424 INFO:tasks.rados.rados.0.smithi071.stdout:1154: expect deleted 2023-12-15T23:01:07.424 INFO:tasks.rados.rados.0.smithi071.stdout:1155: write (excl) oid 39 current snap is 131 2023-12-15T23:01:07.424 INFO:tasks.rados.rados.0.smithi071.stdout:1146: finishing write tid 5 to smithi07183189-15 2023-12-15T23:01:07.424 INFO:tasks.rados.rados.0.smithi071.stdout:1146: oid 15 updating version 0 to 268 2023-12-15T23:01:07.424 INFO:tasks.rados.rados.0.smithi071.stdout:1146: oid 15 version 268 is already newer than 266 2023-12-15T23:01:07.424 INFO:tasks.rados.rados.0.smithi071.stdout:1146: oid 15 version 268 is already newer than 265 2023-12-15T23:01:07.424 INFO:tasks.rados.rados.0.smithi071.stdout:1146: oid 15 version 268 is already newer than 268 2023-12-15T23:01:07.424 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 15 v 268 (ObjNum 555 snap 131 seq_num 555) dirty exists 2023-12-15T23:01:07.424 INFO:tasks.rados.rados.0.smithi071.stdout:1146: left oid 15 (ObjNum 555 snap 131 seq_num 555) 2023-12-15T23:01:07.424 INFO:tasks.rados.rados.0.smithi071.stdout:1149: finishing write tid 1 to smithi07183189-20 2023-12-15T23:01:07.424 INFO:tasks.rados.rados.0.smithi071.stdout:1141: finishing write tid 3 to smithi07183189-12 2023-12-15T23:01:07.425 INFO:tasks.rados.rados.0.smithi071.stdout:1155: seq_num 560 ranges {611920=633209,1703328=634848,2812170=507386} 2023-12-15T23:01:07.430 INFO:tasks.rados.rados.0.smithi071.stdout:1155: writing smithi07183189-39 from 611920 to 1245129 tid 1 2023-12-15T23:01:07.436 INFO:tasks.rados.rados.0.smithi071.stdout:1155: writing smithi07183189-39 from 1703328 to 2338176 tid 2 2023-12-15T23:01:07.442 INFO:tasks.rados.rados.0.smithi071.stdout:1155: writing smithi07183189-39 from 2812170 to 3319556 tid 3 2023-12-15T23:01:07.442 INFO:tasks.rados.rados.0.smithi071.stdout:1146: done (9 left) 2023-12-15T23:01:07.442 INFO:tasks.rados.rados.0.smithi071.stdout:1156: read oid 1 snap -1 2023-12-15T23:01:07.443 INFO:tasks.rados.rados.0.smithi071.stdout:1156: expect (ObjNum 543 snap 127 seq_num 543) 2023-12-15T23:01:07.443 INFO:tasks.rados.rados.0.smithi071.stdout:1157: write oid 4 current snap is 131 2023-12-15T23:01:07.443 INFO:tasks.rados.rados.0.smithi071.stdout:1157: seq_num 561 ranges {628727=708458,2045457=494859,3123240=305487} 2023-12-15T23:01:07.450 INFO:tasks.rados.rados.0.smithi071.stdout:1157: writing smithi07183189-4 from 628727 to 1337185 tid 1 2023-12-15T23:01:07.452 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T23:01:07.458 INFO:tasks.rados.rados.0.smithi071.stdout:1157: writing smithi07183189-4 from 2045457 to 2540316 tid 2 2023-12-15T23:01:07.464 INFO:tasks.rados.rados.0.smithi071.stdout:1157: writing smithi07183189-4 from 3123240 to 3428727 tid 3 2023-12-15T23:01:07.464 INFO:tasks.rados.rados.0.smithi071.stdout:1158: read oid 26 snap 129 2023-12-15T23:01:07.464 INFO:tasks.rados.rados.0.smithi071.stdout:1158: expect (ObjNum 452 snap 104 seq_num 452) 2023-12-15T23:01:07.464 INFO:tasks.rados.rados.0.smithi071.stdout:1159: snap_remove snap 113 2023-12-15T23:01:07.562 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:01:07.561+0000 7f8dc079e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:01:07.565 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:01:07.565+0000 7f8dc079e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:01:08.408 INFO:tasks.rados.rados.0.smithi071.stdout:1159: done (12 left) 2023-12-15T23:01:08.408 INFO:tasks.rados.rados.0.smithi071.stdout:1160: read oid 15 snap 123 2023-12-15T23:01:08.408 INFO:tasks.rados.rados.0.smithi071.stdout:1160: expect deleted 2023-12-15T23:01:08.408 INFO:tasks.rados.rados.0.smithi071.stdout:1161: write oid 29 current snap is 131 2023-12-15T23:01:08.408 INFO:tasks.rados.rados.0.smithi071.stdout:1144: finishing write tid 3 to smithi07183189-9 2023-12-15T23:01:08.408 INFO:tasks.rados.rados.0.smithi071.stdout:1141: finishing write tid 4 to smithi07183189-12 2023-12-15T23:01:08.408 INFO:tasks.rados.rados.0.smithi071.stdout:1161: seq_num 562 ranges {645534=783707,2103939=638517,3150663=294871} 2023-12-15T23:01:08.416 INFO:tasks.rados.rados.0.smithi071.stdout:1161: writing smithi07183189-29 from 645534 to 1429241 tid 1 2023-12-15T23:01:08.423 INFO:tasks.rados.rados.0.smithi071.stdout:1161: writing smithi07183189-29 from 2103939 to 2742456 tid 2 2023-12-15T23:01:08.427 INFO:tasks.rados.rados.0.smithi071.stdout:1161: writing smithi07183189-29 from 3150663 to 3445534 tid 3 2023-12-15T23:01:08.428 INFO:tasks.rados.rados.0.smithi071.stdout:1162: read oid 40 snap -1 2023-12-15T23:01:08.428 INFO:tasks.rados.rados.0.smithi071.stdout:1162: expect (ObjNum 537 snap 124 seq_num 537) 2023-12-15T23:01:08.428 INFO:tasks.rados.rados.0.smithi071.stdout:1163: read oid 45 snap 129 2023-12-15T23:01:08.428 INFO:tasks.rados.rados.0.smithi071.stdout:1163: expect (ObjNum 514 snap 117 seq_num 514) 2023-12-15T23:01:08.428 INFO:tasks.rados.rados.0.smithi071.stdout: waiting on 16 2023-12-15T23:01:08.428 INFO:tasks.rados.rados.0.smithi071.stdout:1141: finishing write tid 5 to smithi07183189-12 2023-12-15T23:01:08.428 INFO:tasks.rados.rados.0.smithi071.stdout:1141: oid 12 updating version 0 to 191 2023-12-15T23:01:08.428 INFO:tasks.rados.rados.0.smithi071.stdout:1141: oid 12 version 191 is already newer than 191 2023-12-15T23:01:08.428 INFO:tasks.rados.rados.0.smithi071.stdout:1141: oid 12 version 191 is already newer than 189 2023-12-15T23:01:08.428 INFO:tasks.rados.rados.0.smithi071.stdout:1141: oid 12 version 191 is already newer than 188 2023-12-15T23:01:08.428 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 12 v 191 (ObjNum 553 snap 131 seq_num 553) dirty exists 2023-12-15T23:01:08.428 INFO:tasks.rados.rados.0.smithi071.stdout:1141: left oid 12 (ObjNum 553 snap 131 seq_num 553) 2023-12-15T23:01:08.428 INFO:tasks.rados.rados.0.smithi071.stdout:1147: expect (ObjNum 545 snap 128 seq_num 545) 2023-12-15T23:01:08.516 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T23:01:08.516 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T23:01:08.516 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-12-15T23:01:08.517 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-12-15T23:01:08.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:01:08.626+0000 7fdca06ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:01:08.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:01:08.633+0000 7fdca06ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:01:08.646 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T23:01:08.646 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T23:01:08.646 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T23:01:08.646 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T23:01:08.646 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T23:01:08.646 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T23:01:08.646 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T23:01:08.646 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T23:01:08.646 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T23:01:08.646 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T23:01:08.646 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T23:01:08.646 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T23:01:08.649 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T23:01:08.650 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:01:08.650 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-12-15T23:01:08.667 INFO:tasks.rados.rados.0.smithi071.stdout:1144: finishing write tid 4 to smithi07183189-9 2023-12-15T23:01:08.667 INFO:tasks.rados.rados.0.smithi071.stdout:1149: finishing write tid 2 to smithi07183189-20 2023-12-15T23:01:08.667 INFO:tasks.rados.rados.0.smithi071.stdout:1144: finishing write tid 5 to smithi07183189-9 2023-12-15T23:01:08.667 INFO:tasks.rados.rados.0.smithi071.stdout:1144: oid 9 updating version 0 to 328 2023-12-15T23:01:08.668 INFO:tasks.rados.rados.0.smithi071.stdout:1144: oid 9 version 328 is already newer than 327 2023-12-15T23:01:08.668 INFO:tasks.rados.rados.0.smithi071.stdout:1144: oid 9 updating version 328 to 330 2023-12-15T23:01:08.668 INFO:tasks.rados.rados.0.smithi071.stdout:1144: oid 9 version 330 is already newer than 329 2023-12-15T23:01:08.668 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 9 v 330 (ObjNum 554 snap 131 seq_num 554) dirty exists 2023-12-15T23:01:08.668 INFO:tasks.rados.rados.0.smithi071.stdout:1144: left oid 9 (ObjNum 554 snap 131 seq_num 554) 2023-12-15T23:01:08.668 INFO:tasks.rados.rados.0.smithi071.stdout:1148: finishing write tid 1 to smithi07183189-35 2023-12-15T23:01:08.668 INFO:tasks.rados.rados.0.smithi071.stdout:1141: done (15 left) 2023-12-15T23:01:08.668 INFO:tasks.rados.rados.0.smithi071.stdout:1144: done (14 left) 2023-12-15T23:01:08.668 INFO:tasks.rados.rados.0.smithi071.stdout:1147: done (13 left) 2023-12-15T23:01:08.668 INFO:tasks.rados.rados.0.smithi071.stdout:1164: delete oid 10 current snap is 131 2023-12-15T23:01:08.668 INFO:tasks.rados.rados.0.smithi071.stdout:1149: finishing write tid 3 to smithi07183189-20 2023-12-15T23:01:08.668 INFO:tasks.rados.rados.0.smithi071.stdout:1149: finishing write tid 4 to smithi07183189-20 2023-12-15T23:01:08.668 INFO:tasks.rados.rados.0.smithi071.stdout:1149: finishing write tid 5 to smithi07183189-20 2023-12-15T23:01:08.668 INFO:tasks.rados.rados.0.smithi071.stdout:1149: oid 20 updating version 0 to 660 2023-12-15T23:01:08.668 INFO:tasks.rados.rados.0.smithi071.stdout:1149: oid 20 version 660 is already newer than 657 2023-12-15T23:01:08.668 INFO:tasks.rados.rados.0.smithi071.stdout:1149: oid 20 version 660 is already newer than 658 2023-12-15T23:01:08.668 INFO:tasks.rados.rados.0.smithi071.stdout:1149: oid 20 version 660 is already newer than 660 2023-12-15T23:01:08.669 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 20 v 660 (ObjNum 557 snap 131 seq_num 557) dirty exists 2023-12-15T23:01:08.669 INFO:tasks.rados.rados.0.smithi071.stdout:1149: left oid 20 (ObjNum 557 snap 131 seq_num 557) 2023-12-15T23:01:08.669 INFO:tasks.rados.rados.0.smithi071.stdout:1148: finishing write tid 2 to smithi07183189-35 2023-12-15T23:01:08.669 INFO:tasks.rados.rados.0.smithi071.stdout:1148: finishing write tid 3 to smithi07183189-35 2023-12-15T23:01:08.669 INFO:tasks.rados.rados.0.smithi071.stdout:1148: finishing write tid 4 to smithi07183189-35 2023-12-15T23:01:08.669 INFO:tasks.rados.rados.0.smithi071.stdout:1148: finishing write tid 5 to smithi07183189-35 2023-12-15T23:01:08.669 INFO:tasks.rados.rados.0.smithi071.stdout:1148: oid 35 updating version 0 to 332 2023-12-15T23:01:08.669 INFO:tasks.rados.rados.0.smithi071.stdout:1148: oid 35 updating version 332 to 333 2023-12-15T23:01:08.669 INFO:tasks.rados.rados.0.smithi071.stdout:1148: oid 35 updating version 333 to 334 2023-12-15T23:01:08.669 INFO:tasks.rados.rados.0.smithi071.stdout:1148: oid 35 version 334 is already newer than 332 2023-12-15T23:01:08.669 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 35 v 334 (ObjNum 556 snap 131 seq_num 556) dirty exists 2023-12-15T23:01:08.669 INFO:tasks.rados.rados.0.smithi071.stdout:1148: left oid 35 (ObjNum 556 snap 131 seq_num 556) 2023-12-15T23:01:08.669 INFO:tasks.rados.rados.0.smithi071.stdout:1156: expect (ObjNum 543 snap 127 seq_num 543) 2023-12-15T23:01:08.759 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:01:08.754+0000 7f3afbb9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:01:08.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:01:08.762+0000 7f3afbb9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:01:08.774 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T23:01:08.774 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T23:01:08.774 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T23:01:08.774 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T23:01:08.774 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T23:01:08.774 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T23:01:08.774 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T23:01:08.774 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T23:01:08.774 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T23:01:08.774 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T23:01:08.774 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T23:01:08.774 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T23:01:08.776 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T23:01:08.777 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:01:08.777 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-12-15T23:01:08.838 INFO:tasks.rados.rados.0.smithi071.stdout:1161: finishing write tid 1 to smithi07183189-29 2023-12-15T23:01:08.838 INFO:tasks.rados.rados.0.smithi071.stdout:1161: finishing write tid 2 to smithi07183189-29 2023-12-15T23:01:08.838 INFO:tasks.rados.rados.0.smithi071.stdout:1161: finishing write tid 3 to smithi07183189-29 2023-12-15T23:01:08.838 INFO:tasks.rados.rados.0.smithi071.stdout:1161: finishing write tid 4 to smithi07183189-29 2023-12-15T23:01:08.838 INFO:tasks.rados.rados.0.smithi071.stdout:1161: finishing write tid 5 to smithi07183189-29 2023-12-15T23:01:08.839 INFO:tasks.rados.rados.0.smithi071.stdout:1161: oid 29 updating version 0 to 670 2023-12-15T23:01:08.839 INFO:tasks.rados.rados.0.smithi071.stdout:1161: oid 29 version 670 is already newer than 667 2023-12-15T23:01:08.839 INFO:tasks.rados.rados.0.smithi071.stdout:1161: oid 29 version 670 is already newer than 670 2023-12-15T23:01:08.839 INFO:tasks.rados.rados.0.smithi071.stdout:1161: oid 29 version 670 is already newer than 668 2023-12-15T23:01:08.839 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 29 v 670 (ObjNum 562 snap 131 seq_num 562) dirty exists 2023-12-15T23:01:08.839 INFO:tasks.rados.rados.0.smithi071.stdout:1161: left oid 29 (ObjNum 562 snap 131 seq_num 562) 2023-12-15T23:01:08.839 INFO:tasks.rados.rados.0.smithi071.stdout:1148: done (13 left) 2023-12-15T23:01:08.839 INFO:tasks.rados.rados.0.smithi071.stdout:1149: done (12 left) 2023-12-15T23:01:08.839 INFO:tasks.rados.rados.0.smithi071.stdout:1154: done (11 left) 2023-12-15T23:01:08.839 INFO:tasks.rados.rados.0.smithi071.stdout:1156: done (10 left) 2023-12-15T23:01:08.839 INFO:tasks.rados.rados.0.smithi071.stdout:1160: done (9 left) 2023-12-15T23:01:08.839 INFO:tasks.rados.rados.0.smithi071.stdout:1161: done (8 left) 2023-12-15T23:01:08.839 INFO:tasks.rados.rados.0.smithi071.stdout:1164: done (7 left) 2023-12-15T23:01:08.840 INFO:tasks.rados.rados.0.smithi071.stdout:1165: read oid 27 snap -1 2023-12-15T23:01:08.840 INFO:tasks.rados.rados.0.smithi071.stdout:1165: expect (ObjNum 493 snap 113 seq_num 493) 2023-12-15T23:01:08.840 INFO:tasks.rados.rados.0.smithi071.stdout:1166: write oid 49 current snap is 131 2023-12-15T23:01:08.840 INFO:tasks.rados.rados.0.smithi071.stdout:1162: expect (ObjNum 537 snap 124 seq_num 537) 2023-12-15T23:01:08.924 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:01:08.920+0000 7f1f45af0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:01:08.928 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:01:08.927+0000 7f1f45af0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:01:08.940 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T23:01:08.940 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T23:01:08.940 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T23:01:08.940 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T23:01:08.940 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T23:01:08.940 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T23:01:08.940 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T23:01:08.940 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T23:01:08.940 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T23:01:08.940 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T23:01:08.940 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T23:01:08.940 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T23:01:08.942 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T23:01:08.943 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:01:08.944 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-12-15T23:01:09.003 INFO:tasks.rados.rados.0.smithi071.stdout:1163: expect (ObjNum 514 snap 117 seq_num 514) 2023-12-15T23:01:09.091 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:01:09.086+0000 7fdae106d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:01:09.093 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:01:09.093+0000 7fdae106d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:01:09.164 INFO:tasks.rados.rados.0.smithi071.stdout:1166: seq_num 564 ranges {679148=650558,1937256=642186,3321862=157286} 2023-12-15T23:01:09.171 INFO:tasks.rados.rados.0.smithi071.stdout:1166: writing smithi07183189-49 from 679148 to 1329706 tid 1 2023-12-15T23:01:09.178 INFO:tasks.rados.rados.0.smithi071.stdout:1166: writing smithi07183189-49 from 1937256 to 2579442 tid 2 2023-12-15T23:01:09.183 INFO:tasks.rados.rados.0.smithi071.stdout:1166: writing smithi07183189-49 from 3321862 to 3479148 tid 3 2023-12-15T23:01:09.183 INFO:tasks.rados.rados.0.smithi071.stdout:1162: done (8 left) 2023-12-15T23:01:09.183 INFO:tasks.rados.rados.0.smithi071.stdout:1163: done (7 left) 2023-12-15T23:01:09.183 INFO:tasks.rados.rados.0.smithi071.stdout:1167: write (excl) oid 50 current snap is 131 2023-12-15T23:01:09.183 INFO:tasks.rados.rados.0.smithi071.stdout:1167: seq_num 565 ranges {695955=725807,1879385=502197,2949285=480099} 2023-12-15T23:01:09.190 INFO:tasks.rados.rados.0.smithi071.stdout:1167: writing smithi07183189-50 from 695955 to 1421762 tid 1 2023-12-15T23:01:09.196 INFO:tasks.rados.rados.0.smithi071.stdout:1167: writing smithi07183189-50 from 1879385 to 2381582 tid 2 2023-12-15T23:01:09.204 INFO:tasks.rados.rados.0.smithi071.stdout:1167: writing smithi07183189-50 from 2949285 to 3429384 tid 3 2023-12-15T23:01:09.204 INFO:tasks.rados.rados.0.smithi071.stdout:1168: delete oid 15 current snap is 131 2023-12-15T23:01:09.210 INFO:tasks.rados.rados.0.smithi071.stdout:1167: finishing write tid 1 to smithi07183189-50 2023-12-15T23:01:09.229 INFO:tasks.rados.rados.0.smithi071.stdout:1167: finishing write tid 2 to smithi07183189-50 2023-12-15T23:01:09.246 INFO:tasks.rados.rados.0.smithi071.stdout:1167: finishing write tid 3 to smithi07183189-50 2023-12-15T23:01:09.247 INFO:tasks.rados.rados.0.smithi071.stdout:1167: finishing write tid 4 to smithi07183189-50 2023-12-15T23:01:09.248 INFO:tasks.rados.rados.0.smithi071.stdout:1167: finishing write tid 5 to smithi07183189-50 2023-12-15T23:01:09.248 INFO:tasks.rados.rados.0.smithi071.stdout:1167: oid 50 updating version 0 to 678 2023-12-15T23:01:09.248 INFO:tasks.rados.rados.0.smithi071.stdout:1167: oid 50 version 678 is already newer than 676 2023-12-15T23:01:09.249 INFO:tasks.rados.rados.0.smithi071.stdout:1167: oid 50 version 678 is already newer than 678 2023-12-15T23:01:09.249 INFO:tasks.rados.rados.0.smithi071.stdout:1167: oid 50 version 678 is already newer than 675 2023-12-15T23:01:09.249 INFO:tasks.rados.rados.0.smithi071.stdout:update_object_version oid 50 v 678 (ObjNum 565 snap 131 seq_num 565) dirty exists 2023-12-15T23:01:09.249 INFO:tasks.rados.rados.0.smithi071.stdout:1167: left oid 50 (ObjNum 565 snap 131 seq_num 565) 2023-12-15T23:01:14.275 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2023-12-15T23:01:19.780 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~6s 2023-12-15T23:01:25.285 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~11s 2023-12-15T23:01:30.788 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~17s 2023-12-15T23:01:35.319 INFO:teuthology.orchestra.run.smithi071.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-12-15T23:01:35.319 INFO:teuthology.orchestra.run.smithi071.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-12-15T23:01:35.324 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-12-15T23:01:35.324 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-15T23:01:35.324 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-12-15T23:01:35.324 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:01:36.091 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-12-15T23:01:36.091 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-12-15T23:01:36.091 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~22s 2023-12-15T23:01:37.325 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T23:01:37.434 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:01:37.433+0000 7f27e48ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:01:37.437 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:01:37.437+0000 7f27e48ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:01:40.325 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T23:01:40.326 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T23:01:40.326 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-12-15T23:01:40.326 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-12-15T23:01:40.444 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:01:40.439+0000 7f816ff15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:01:40.447 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:01:40.447+0000 7f816ff15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:01:41.393 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~5s 2023-12-15T23:01:41.394 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~5s 2023-12-15T23:01:41.394 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~27s 2023-12-15T23:01:46.696 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~11s 2023-12-15T23:01:46.696 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~11s 2023-12-15T23:01:46.696 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~33s 2023-12-15T23:01:52.000 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~16s 2023-12-15T23:01:52.000 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~16s 2023-12-15T23:01:52.000 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~38s 2023-12-15T23:01:57.302 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~21s 2023-12-15T23:01:57.302 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~21s 2023-12-15T23:01:57.302 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~43s 2023-12-15T23:02:02.605 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~27s 2023-12-15T23:02:02.605 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~27s 2023-12-15T23:02:02.605 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~49s 2023-12-15T23:02:07.907 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~32s 2023-12-15T23:02:07.908 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~32s 2023-12-15T23:02:07.908 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~54s 2023-12-15T23:02:13.211 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~37s 2023-12-15T23:02:13.211 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~37s 2023-12-15T23:02:13.211 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~59s 2023-12-15T23:02:18.513 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~42s 2023-12-15T23:02:18.514 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~42s 2023-12-15T23:02:18.514 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~64s 2023-12-15T23:02:23.816 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~48s 2023-12-15T23:02:23.816 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~48s 2023-12-15T23:02:23.816 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~70s 2023-12-15T23:02:29.119 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~53s 2023-12-15T23:02:29.119 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~53s 2023-12-15T23:02:29.119 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~75s 2023-12-15T23:02:34.421 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~58s 2023-12-15T23:02:34.422 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~58s 2023-12-15T23:02:34.422 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~80s 2023-12-15T23:02:37.302 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:02:37.301+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:02:39.725 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~64s 2023-12-15T23:02:39.725 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~64s 2023-12-15T23:02:39.726 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~86s 2023-12-15T23:02:42.302 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:02:42.302+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:02:45.028 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~69s 2023-12-15T23:02:45.028 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~69s 2023-12-15T23:02:45.028 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~91s 2023-12-15T23:02:47.303 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:02:47.302+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:02:50.331 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~74s 2023-12-15T23:02:50.332 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~74s 2023-12-15T23:02:50.332 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~96s 2023-12-15T23:02:52.305 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:02:52.305+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:02:55.635 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~80s 2023-12-15T23:02:55.635 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~80s 2023-12-15T23:02:55.635 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~102s 2023-12-15T23:02:57.306 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:02:57.306+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:03:00.938 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~85s 2023-12-15T23:03:00.938 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~85s 2023-12-15T23:03:00.938 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~107s 2023-12-15T23:03:02.306 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:03:02.306+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:03:06.241 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~90s 2023-12-15T23:03:06.241 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~90s 2023-12-15T23:03:06.241 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~112s 2023-12-15T23:03:07.308 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:03:07.307+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:03:11.544 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~95s 2023-12-15T23:03:11.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~95s 2023-12-15T23:03:11.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~117s 2023-12-15T23:03:12.309 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:03:12.309+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:03:16.847 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~101s 2023-12-15T23:03:16.847 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~101s 2023-12-15T23:03:16.848 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~123s 2023-12-15T23:03:17.312 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:03:17.309+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:03:22.151 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~106s 2023-12-15T23:03:22.151 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~106s 2023-12-15T23:03:22.151 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~128s 2023-12-15T23:03:22.311 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:03:22.310+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:03:27.311 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:03:27.311+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:03:27.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~111s 2023-12-15T23:03:27.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~111s 2023-12-15T23:03:27.454 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~133s 2023-12-15T23:03:32.313 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:03:32.312+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:03:32.756 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~117s 2023-12-15T23:03:32.756 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~117s 2023-12-15T23:03:32.756 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~139s 2023-12-15T23:03:37.314 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:03:37.313+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:03:37.362 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-15T23:03:37.362 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-15T23:03:38.059 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~122s 2023-12-15T23:03:38.059 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~122s 2023-12-15T23:03:38.059 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~144s 2023-12-15T23:03:39.364 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T23:03:39.479 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:03:39.477+0000 7fd5b686d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:03:39.483 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:03:39.482+0000 7fd5b686d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:03:40.363 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-15T23:03:40.364 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:03:40.364 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-12-15T23:03:40.481 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:03:40.476+0000 7f8e0462d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:03:40.484 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:03:40.483+0000 7f8e0462d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:03:42.314 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:03:42.314+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:03:43.362 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~127s 2023-12-15T23:03:43.362 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~127s 2023-12-15T23:03:43.362 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~149s 2023-12-15T23:03:47.315 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:03:47.315+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:03:48.665 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~133s 2023-12-15T23:03:48.665 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~133s 2023-12-15T23:03:48.665 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~155s 2023-12-15T23:03:52.316 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:03:52.316+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:03:53.968 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~138s 2023-12-15T23:03:53.968 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~138s 2023-12-15T23:03:53.968 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~160s 2023-12-15T23:03:57.318 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:03:57.317+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:03:59.271 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~143s 2023-12-15T23:03:59.271 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~143s 2023-12-15T23:03:59.271 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~165s 2023-12-15T23:04:02.318 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:04:02.318+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:04:04.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~148s 2023-12-15T23:04:04.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~148s 2023-12-15T23:04:04.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~170s 2023-12-15T23:04:07.319 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:04:07.319+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:04:09.877 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~154s 2023-12-15T23:04:09.877 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~154s 2023-12-15T23:04:09.877 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~176s 2023-12-15T23:04:12.321 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:04:12.321+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:04:15.180 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~159s 2023-12-15T23:04:15.180 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~159s 2023-12-15T23:04:15.180 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~181s 2023-12-15T23:04:17.322 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:04:17.321+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:04:20.483 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~164s 2023-12-15T23:04:20.483 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~164s 2023-12-15T23:04:20.483 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~186s 2023-12-15T23:04:22.324 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:04:22.323+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:04:25.785 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~170s 2023-12-15T23:04:25.785 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~170s 2023-12-15T23:04:25.785 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~192s 2023-12-15T23:04:27.324 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:04:27.324+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:04:31.088 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~175s 2023-12-15T23:04:31.088 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~175s 2023-12-15T23:04:31.088 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~197s 2023-12-15T23:04:32.326 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:04:32.324+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:04:36.391 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~180s 2023-12-15T23:04:36.391 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~180s 2023-12-15T23:04:36.391 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~202s 2023-12-15T23:04:37.327 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:04:37.326+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:04:41.694 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~186s 2023-12-15T23:04:41.694 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~186s 2023-12-15T23:04:41.694 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~208s 2023-12-15T23:04:42.328 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:04:42.327+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:04:46.997 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~191s 2023-12-15T23:04:46.997 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~191s 2023-12-15T23:04:46.997 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~213s 2023-12-15T23:04:47.328 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:04:47.328+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:04:52.299 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~196s 2023-12-15T23:04:52.299 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~196s 2023-12-15T23:04:52.299 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~218s 2023-12-15T23:04:52.330 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:04:52.330+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:04:57.331 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:04:57.331+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:04:57.601 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~202s 2023-12-15T23:04:57.601 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~202s 2023-12-15T23:04:57.601 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~224s 2023-12-15T23:05:02.332 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:05:02.332+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:05:02.904 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~207s 2023-12-15T23:05:02.904 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~207s 2023-12-15T23:05:02.904 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~229s 2023-12-15T23:05:07.333 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:05:07.333+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:05:08.207 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~212s 2023-12-15T23:05:08.207 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~212s 2023-12-15T23:05:08.208 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~234s 2023-12-15T23:05:12.336 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:05:12.334+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:05:13.510 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~217s 2023-12-15T23:05:13.510 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~217s 2023-12-15T23:05:13.510 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~239s 2023-12-15T23:05:17.351 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:05:17.335+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:05:18.814 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~223s 2023-12-15T23:05:18.814 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~223s 2023-12-15T23:05:18.814 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~245s 2023-12-15T23:05:22.336 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:05:22.336+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:05:24.116 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~228s 2023-12-15T23:05:24.117 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~228s 2023-12-15T23:05:24.117 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~250s 2023-12-15T23:05:27.338 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:05:27.336+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:05:29.420 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~233s 2023-12-15T23:05:29.421 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~233s 2023-12-15T23:05:29.421 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~255s 2023-12-15T23:05:32.339 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:05:32.338+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:05:34.723 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~239s 2023-12-15T23:05:34.723 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~239s 2023-12-15T23:05:34.724 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~261s 2023-12-15T23:05:37.340 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:05:37.339+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:05:39.402 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-15T23:05:39.403 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-15T23:05:40.026 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~244s 2023-12-15T23:05:40.026 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~244s 2023-12-15T23:05:40.026 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~266s 2023-12-15T23:05:40.401 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-15T23:05:40.401 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:05:40.402 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-12-15T23:05:40.519 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:05:40.514+0000 7fc25ad9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:05:40.521 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:05:40.521+0000 7fc25ad9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:05:40.535 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2023-12-15T23:05:40.535 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2023-12-15T23:05:40.535 INFO:teuthology.orchestra.run.smithi071.stderr:config help 2023-12-15T23:05:40.536 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2023-12-15T23:05:40.536 INFO:teuthology.orchestra.run.smithi071.stderr:config show 2023-12-15T23:05:40.536 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_ops 2023-12-15T23:05:40.536 INFO:teuthology.orchestra.run.smithi071.stderr:dump_historic_slow_ops 2023-12-15T23:05:40.536 INFO:teuthology.orchestra.run.smithi071.stderr:dump_metrics [] 2023-12-15T23:05:40.536 INFO:teuthology.orchestra.run.smithi071.stderr:dump_ops_in_flight 2023-12-15T23:05:40.536 INFO:teuthology.orchestra.run.smithi071.stderr:dump_pgstate_history 2023-12-15T23:05:40.536 INFO:teuthology.orchestra.run.smithi071.stderr:dump_recovery_reservations 2023-12-15T23:05:40.536 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: invalid command 2023-12-15T23:05:40.538 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-12-15T23:05:40.539 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:05:40.539 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-12-15T23:05:40.645 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:05:40.640+0000 7fbd9115e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:05:40.647 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:05:40.647+0000 7fbd9115e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:05:40.651 INFO:teuthology.orchestra.run.smithi071.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-12-15T23:05:40.654 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-12-15T23:05:40.654 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:05:41.404 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T23:05:41.510 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:05:41.508+0000 7f92eb11e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:05:41.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:05:41.513+0000 7f92eb11e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:05:42.340 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:05:42.340+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:05:45.330 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~249s 2023-12-15T23:05:45.330 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~249s 2023-12-15T23:05:45.330 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~271s 2023-12-15T23:05:45.655 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T23:05:45.655 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T23:05:45.655 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2023-12-15T23:05:45.655 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2023-12-15T23:05:45.774 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:05:45.769+0000 7fb22671e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:05:45.777 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:05:45.776+0000 7fb22671e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:05:47.342 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:05:47.341+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:05:50.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~255s 2023-12-15T23:05:50.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~255s 2023-12-15T23:05:50.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~277s 2023-12-15T23:05:52.343 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:05:52.342+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:05:55.936 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~260s 2023-12-15T23:05:55.937 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~260s 2023-12-15T23:05:55.937 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~282s 2023-12-15T23:05:57.344 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:05:57.343+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:06:01.239 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~265s 2023-12-15T23:06:01.239 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~265s 2023-12-15T23:06:01.239 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~287s 2023-12-15T23:06:02.345 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:06:02.344+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:06:06.542 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~270s 2023-12-15T23:06:06.542 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~270s 2023-12-15T23:06:06.542 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~292s 2023-12-15T23:06:07.346 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:06:07.345+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:06:11.846 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~276s 2023-12-15T23:06:11.846 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~276s 2023-12-15T23:06:11.846 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~298s 2023-12-15T23:06:12.347 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:06:12.347+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:06:17.149 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~281s 2023-12-15T23:06:17.150 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~281s 2023-12-15T23:06:17.150 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~303s 2023-12-15T23:06:17.150 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-12-15T23:06:17.348 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:06:17.347+0000 7f92785f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.71:6801/1516130097 for 29sec e282 v282) 2023-12-15T23:06:17.452 INFO:tasks.ceph.osd.2:Sent signal 15 2023-12-15T23:06:17.453 INFO:tasks.ceph.mon.a:Sent signal 15 2023-12-15T23:06:17.453 INFO:tasks.ceph.mgr.x:Sent signal 15 2023-12-15T23:06:17.454 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:06:17.453+0000 7f927b5fc640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 81391) UID: 0 2023-12-15T23:06:17.454 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-12-15T23:06:17.453+0000 7f927b5fc640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-12-15T23:07:41.439 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-15T23:07:41.440 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-15T23:07:43.442 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T23:07:43.557 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:07:43.555+0000 7fdcf5b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:07:45.693 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-15T23:07:45.693 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:07:45.693 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2023-12-15T23:07:45.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:07:45.801+0000 7f850875e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:09:43.479 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-15T23:09:43.480 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-15T23:09:45.481 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T23:09:45.599 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:09:45.596+0000 7f8f34a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:09:45.727 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-15T23:09:45.728 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:09:45.728 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2023-12-15T23:09:45.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:09:45.830+0000 7f0a71115640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:11:45.519 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-15T23:11:45.520 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-15T23:11:45.760 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-15T23:11:45.761 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:11:45.761 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2023-12-15T23:11:45.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:11:45.874+0000 7f48e9b00640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:11:47.522 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T23:11:47.638 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:11:47.636+0000 7f24be19e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:13:45.799 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-15T23:13:45.800 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2023-12-15T23:13:47.560 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-15T23:13:47.560 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-15T23:13:49.562 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T23:13:49.680 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:13:49.678+0000 7fe65fc13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:13:50.802 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2023-12-15T23:13:50.802 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2023-12-15T23:13:50.802 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-15T23:13:50.921 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:13:50.916+0000 7f7ce2b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:15:49.602 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-15T23:15:49.603 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-15T23:15:50.840 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-15T23:15:50.843 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_cd750a7fc894b5dea3d055d595c0da884f63f94f/qa/tasks/ceph_manager.py", line 190, in wrapper return func(self) File "/home/teuthworker/src/github.com_ceph_ceph-c_cd750a7fc894b5dea3d055d595c0da884f63f94f/qa/tasks/ceph_manager.py", line 1417, in _do_thrash self.choose_action()() File "/home/teuthworker/src/github.com_ceph_ceph-c_cd750a7fc894b5dea3d055d595c0da884f63f94f/qa/tasks/ceph_manager.py", line 779, in force_cancel_recovery self.force_recovery() File "/home/teuthworker/src/github.com_ceph_ceph-c_cd750a7fc894b5dea3d055d595c0da884f63f94f/qa/tasks/ceph_manager.py", line 748, in force_recovery j = self.ceph_manager.get_pgids_to_force(backfill) File "/home/teuthworker/src/github.com_ceph_ceph-c_cd750a7fc894b5dea3d055d595c0da884f63f94f/qa/tasks/ceph_manager.py", line 2385, in get_pgids_to_force j = self.get_pg_stats(); File "/home/teuthworker/src/github.com_ceph_ceph-c_cd750a7fc894b5dea3d055d595c0da884f63f94f/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_cd750a7fc894b5dea3d055d595c0da884f63f94f/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_cd750a7fc894b5dea3d055d595c0da884f63f94f/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi071 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-12-15T23:15:50.843 ERROR:tasks.thrashosds.thrasher:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_cd750a7fc894b5dea3d055d595c0da884f63f94f/qa/tasks/ceph_manager.py", line 1285, in do_thrash self._do_thrash() File "/home/teuthworker/src/github.com_ceph_ceph-c_cd750a7fc894b5dea3d055d595c0da884f63f94f/qa/tasks/ceph_manager.py", line 190, in wrapper return func(self) File "/home/teuthworker/src/github.com_ceph_ceph-c_cd750a7fc894b5dea3d055d595c0da884f63f94f/qa/tasks/ceph_manager.py", line 1417, in _do_thrash self.choose_action()() File "/home/teuthworker/src/github.com_ceph_ceph-c_cd750a7fc894b5dea3d055d595c0da884f63f94f/qa/tasks/ceph_manager.py", line 779, in force_cancel_recovery self.force_recovery() File "/home/teuthworker/src/github.com_ceph_ceph-c_cd750a7fc894b5dea3d055d595c0da884f63f94f/qa/tasks/ceph_manager.py", line 748, in force_recovery j = self.ceph_manager.get_pgids_to_force(backfill) File "/home/teuthworker/src/github.com_ceph_ceph-c_cd750a7fc894b5dea3d055d595c0da884f63f94f/qa/tasks/ceph_manager.py", line 2385, in get_pgids_to_force j = self.get_pg_stats(); File "/home/teuthworker/src/github.com_ceph_ceph-c_cd750a7fc894b5dea3d055d595c0da884f63f94f/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_cd750a7fc894b5dea3d055d595c0da884f63f94f/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_cd750a7fc894b5dea3d055d595c0da884f63f94f/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi071 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-12-15T23:15:51.605 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T23:15:51.722 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:15:51.719+0000 7f6e7539e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:17:51.644 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-15T23:17:51.645 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-15T23:17:53.647 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T23:17:53.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:17:53.759+0000 7fef52e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:19:53.684 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-15T23:19:53.685 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-15T23:19:55.687 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T23:19:55.802 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:19:55.799+0000 7f412811e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:21:55.724 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-15T23:21:55.725 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-15T23:21:57.727 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T23:21:57.843 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:21:57.840+0000 7fd62b86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:23:57.765 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-15T23:23:57.766 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-15T23:23:59.767 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T23:23:59.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:23:59.880+0000 7f778da6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:25:59.806 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-15T23:25:59.807 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-15T23:26:01.808 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T23:26:01.924 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:26:01.921+0000 7fbee7206640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:28:01.848 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-15T23:28:01.849 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-15T23:28:03.850 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T23:28:03.967 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:28:03.964+0000 7f5bdd79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:30:03.888 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-15T23:30:03.889 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-15T23:30:05.891 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T23:30:06.006 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:30:06.003+0000 7fe1dbc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:32:05.933 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-15T23:32:05.938 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-15T23:32:07.940 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T23:32:08.056 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:32:08.053+0000 7fdf07ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:34:07.978 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-15T23:34:07.979 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-15T23:34:09.981 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T23:34:10.097 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:34:10.094+0000 7fb3ad35e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:36:10.020 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-15T23:36:10.022 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-15T23:36:12.023 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T23:36:12.139 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:36:12.136+0000 7f6f58b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:38:12.061 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-15T23:38:12.062 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-15T23:38:14.063 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T23:38:14.179 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:38:14.175+0000 7f341282d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:40:14.101 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-15T23:40:14.103 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-15T23:40:16.104 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T23:40:16.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:40:16.216+0000 7ffbbbead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:42:16.142 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-15T23:42:16.143 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-15T23:42:18.144 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T23:42:18.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:42:18.257+0000 7fe382b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:44:18.182 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-15T23:44:18.183 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-15T23:44:20.184 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T23:44:20.300 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:44:20.297+0000 7f27964ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:46:20.222 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-15T23:46:20.223 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-15T23:46:22.225 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T23:46:22.339 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:46:22.337+0000 7f17c9a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:48:22.263 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-15T23:48:22.265 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-15T23:48:24.267 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T23:48:24.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:48:24.381+0000 7f59612ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:50:24.305 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-15T23:50:24.306 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-15T23:50:26.307 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T23:50:26.424 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:50:26.422+0000 7f8f346ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:52:26.344 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-15T23:52:26.345 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-15T23:52:28.346 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T23:52:28.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:52:28.459+0000 7f0d06224640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:54:28.384 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-15T23:54:28.385 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-15T23:54:30.385 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T23:54:30.500 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:54:30.498+0000 7f2ca6af0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:56:30.423 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-15T23:56:30.425 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-15T23:56:32.426 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-15T23:56:32.542 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:56:32.540+0000 7f6958aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-15T23:58:32.465 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-15T23:58:32.466 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-15T23:58:34.468 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-15T23:58:34.583 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-15T23:58:34.582+0000 7f5ed819e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T00:00:34.507 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T00:00:34.508 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T00:00:36.509 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T00:00:36.628 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T00:00:36.626+0000 7f040446d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T00:02:36.550 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T00:02:36.551 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T00:02:38.552 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T00:02:38.670 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T00:02:38.669+0000 7f796dc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T00:04:38.594 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T00:04:38.595 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T00:04:40.597 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T00:04:40.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T00:04:40.710+0000 7f919cd5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T00:06:40.635 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T00:06:40.636 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T00:06:42.638 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T00:06:42.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T00:06:42.753+0000 7ff8c859e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T00:08:42.675 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T00:08:42.676 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T00:08:44.678 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T00:08:44.793 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T00:08:44.792+0000 7f3d70b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T00:10:44.717 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T00:10:44.718 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T00:10:46.721 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T00:10:46.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T00:10:46.835+0000 7f385cd5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T00:12:46.760 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T00:12:46.762 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T00:12:48.763 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T00:12:48.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T00:12:48.877+0000 7f1a4979e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T00:14:48.801 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T00:14:48.802 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T00:14:50.803 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T00:14:50.919 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T00:14:50.917+0000 7f386839e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T00:16:50.842 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T00:16:50.843 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T00:16:52.845 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T00:16:52.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T00:16:52.958+0000 7fd93119e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T00:18:52.883 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T00:18:52.884 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T00:18:54.885 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T00:18:55.001 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T00:18:55.000+0000 7ff16059e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T00:20:54.924 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T00:20:54.926 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T00:20:56.927 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T00:20:57.043 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T00:20:57.041+0000 7fd0ecaad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T00:22:56.969 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T00:22:56.970 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T00:22:58.971 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T00:22:59.089 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T00:22:59.084+0000 7f86f70f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T00:24:59.011 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T00:24:59.012 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T00:25:01.013 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T00:25:01.129 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T00:25:01.127+0000 7fbd6535e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T00:27:01.056 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T00:27:01.057 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T00:27:03.058 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T00:27:03.173 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T00:27:03.171+0000 7f2f8319e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T00:29:03.096 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T00:29:03.097 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T00:29:05.100 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T00:29:05.216 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T00:29:05.214+0000 7fd2d662d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T00:31:05.138 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T00:31:05.139 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T00:31:07.141 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T00:31:07.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T00:31:07.254+0000 7f1beb0f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T00:33:07.179 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T00:33:07.180 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T00:33:09.182 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T00:33:09.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T00:33:09.296+0000 7f2a3bd55640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T00:35:09.221 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T00:35:09.222 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T00:35:11.224 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T00:35:11.339 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T00:35:11.337+0000 7f3b8bb9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T00:37:11.262 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T00:37:11.263 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T00:37:13.264 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T00:37:13.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T00:37:13.378+0000 7fb93df9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T00:39:13.302 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T00:39:13.304 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T00:39:15.305 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T00:39:15.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T00:39:15.419+0000 7f52cb59e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T00:41:15.343 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T00:41:15.344 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T00:41:17.345 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T00:41:17.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T00:41:17.458+0000 7f7474f5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T00:43:17.383 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T00:43:17.384 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T00:43:19.386 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T00:43:19.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T00:43:19.500+0000 7fe067d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T00:45:19.424 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T00:45:19.425 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T00:45:21.426 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T00:45:21.540 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T00:45:21.538+0000 7faea6a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T00:47:21.464 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T00:47:21.465 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T00:47:23.467 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T00:47:23.583 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T00:47:23.581+0000 7f8f1d19e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T00:49:23.505 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T00:49:23.506 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T00:49:25.507 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T00:49:25.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T00:49:25.622+0000 7f3961c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T00:51:25.545 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T00:51:25.547 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T00:51:27.548 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T00:51:27.663 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T00:51:27.661+0000 7f309d06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T00:53:27.586 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T00:53:27.588 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T00:53:29.589 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T00:53:29.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T00:53:29.702+0000 7fcb1fede640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T00:55:29.627 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T00:55:29.628 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T00:55:31.630 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T00:55:31.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T00:55:31.744+0000 7fb56b79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T00:57:31.668 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T00:57:31.669 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T00:57:33.671 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T00:57:33.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T00:57:33.785+0000 7ff53fa2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T00:59:33.708 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T00:59:33.709 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T00:59:35.710 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T00:59:35.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T00:59:35.824+0000 7f428ba6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T01:01:35.748 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T01:01:35.750 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T01:01:37.751 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T01:01:37.867 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T01:01:37.865+0000 7f5307555640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T01:03:37.790 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T01:03:37.791 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T01:03:39.792 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T01:03:39.908 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T01:03:39.906+0000 7fb199af7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T01:05:39.830 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T01:05:39.831 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T01:05:41.832 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T01:05:41.947 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T01:05:41.946+0000 7fef5f300640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T01:07:41.870 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T01:07:41.871 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T01:07:43.872 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T01:07:43.987 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T01:07:43.985+0000 7fa3cc11e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T01:09:43.910 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T01:09:43.911 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T01:09:45.912 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T01:09:46.026 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T01:09:46.025+0000 7f8597b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T01:11:45.950 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T01:11:45.951 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T01:11:47.952 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T01:11:48.068 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T01:11:48.067+0000 7f6f23ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T01:13:47.991 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T01:13:47.992 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T01:13:49.993 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T01:13:50.109 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T01:13:50.107+0000 7f05ccb9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T01:15:50.031 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T01:15:50.032 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T01:15:52.033 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T01:15:52.148 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T01:15:52.147+0000 7f6a2375e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T01:17:52.071 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T01:17:52.072 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T01:17:54.073 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T01:17:54.190 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T01:17:54.188+0000 7f7335511640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T01:19:54.111 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T01:19:54.112 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T01:19:56.113 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T01:19:56.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T01:19:56.229+0000 7f48cfb5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T01:21:56.151 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T01:21:56.152 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T01:21:58.153 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T01:21:58.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T01:21:58.268+0000 7f06dbe2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T01:23:58.191 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T01:23:58.192 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T01:24:00.193 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T01:24:00.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T01:24:00.307+0000 7fa73095e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T01:26:00.232 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T01:26:00.233 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T01:26:02.234 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T01:26:02.349 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T01:26:02.347+0000 7f6fdad1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T01:28:02.271 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T01:28:02.272 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T01:28:04.273 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T01:28:04.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T01:28:04.389+0000 7fa54162d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T01:30:04.314 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T01:30:04.315 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T01:30:06.317 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T01:30:06.432 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T01:30:06.430+0000 7fec4002d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T01:32:06.355 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T01:32:06.356 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T01:32:08.357 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T01:32:08.472 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T01:32:08.469+0000 7f3121e13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T01:34:08.395 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T01:34:08.396 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T01:34:10.397 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T01:34:10.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T01:34:10.510+0000 7f56e58ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T01:36:10.435 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T01:36:10.436 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T01:36:12.437 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T01:36:12.553 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T01:36:12.550+0000 7f57c7555640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T01:38:12.476 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T01:38:12.477 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T01:38:14.478 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T01:38:14.594 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T01:38:14.590+0000 7f58c351e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T01:40:14.516 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T01:40:14.517 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T01:40:16.518 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T01:40:16.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T01:40:16.631+0000 7f6deed15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T01:42:16.557 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T01:42:16.558 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T01:42:18.559 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T01:42:18.674 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T01:42:18.671+0000 7f1683ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T01:44:18.597 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T01:44:18.598 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T01:44:20.599 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T01:44:20.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T01:44:20.714+0000 7ffa1ef9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T01:46:20.638 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T01:46:20.639 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T01:46:22.640 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T01:46:22.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T01:46:22.753+0000 7f5055ede640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T01:48:22.678 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T01:48:22.679 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T01:48:24.680 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T01:48:24.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T01:48:24.791+0000 7f768571e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T01:50:24.719 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T01:50:24.720 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T01:50:26.721 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T01:50:26.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T01:50:26.836+0000 7f1b46d1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T01:52:26.760 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T01:52:26.761 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T01:52:28.762 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T01:52:28.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T01:52:28.876+0000 7f4235715640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T01:54:28.801 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T01:54:28.802 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T01:54:30.802 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T01:54:30.918 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T01:54:30.915+0000 7efc0ae2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T01:56:30.841 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T01:56:30.842 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T01:56:32.843 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T01:56:32.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T01:56:32.955+0000 7f2f9991e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T01:58:32.881 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T01:58:32.882 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T01:58:34.883 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T01:58:34.999 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T01:58:34.996+0000 7f98b6cde640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T02:00:34.920 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T02:00:34.921 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T02:00:36.922 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T02:00:37.038 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T02:00:37.035+0000 7f5d7615e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T02:02:36.960 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T02:02:36.961 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T02:02:38.963 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T02:02:39.079 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T02:02:39.076+0000 7f712862d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T02:04:39.003 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T02:04:39.003 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T02:04:41.005 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T02:04:41.120 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T02:04:41.117+0000 7fb9ab42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T02:06:41.043 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T02:06:41.044 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T02:06:43.045 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T02:06:43.161 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T02:06:43.158+0000 7fee56b1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T02:08:43.082 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T02:08:43.083 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T02:08:45.084 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T02:08:45.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T02:08:45.197+0000 7fc007f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T02:10:45.122 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T02:10:45.123 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T02:10:47.124 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T02:10:47.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T02:10:47.239+0000 7f251ccde640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T02:12:47.164 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T02:12:47.164 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T02:12:49.166 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T02:12:49.283 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T02:12:49.280+0000 7fef2f95e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T02:14:49.204 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T02:14:49.205 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T02:14:51.206 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T02:14:51.321 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T02:14:51.318+0000 7fc899d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T02:16:51.243 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T02:16:51.244 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T02:16:53.245 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T02:16:53.360 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T02:16:53.357+0000 7f65def1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T02:18:53.283 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T02:18:53.284 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T02:18:55.286 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T02:18:55.400 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T02:18:55.396+0000 7fae8446d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T02:20:55.324 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T02:20:55.325 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T02:20:57.329 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T02:20:57.445 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T02:20:57.442+0000 7f6eacd1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T02:22:57.366 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T02:22:57.367 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T02:22:59.368 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T02:22:59.484 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T02:22:59.481+0000 7fa87266d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T02:24:59.406 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T02:24:59.407 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T02:25:01.408 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T02:25:01.525 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T02:25:01.522+0000 7fabe162d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T02:27:01.447 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T02:27:01.448 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T02:27:03.449 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T02:27:03.565 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T02:27:03.562+0000 7feb2342d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T02:29:03.487 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T02:29:03.488 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T02:29:05.489 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T02:29:05.604 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T02:29:05.601+0000 7ff39a46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T02:31:05.527 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T02:31:05.528 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T02:31:07.529 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T02:31:07.644 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T02:31:07.642+0000 7f080b100640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T02:33:07.567 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T02:33:07.568 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T02:33:09.569 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T02:33:09.685 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T02:33:09.682+0000 7fc03159e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T02:35:09.607 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T02:35:09.608 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T02:35:11.608 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T02:35:11.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T02:35:11.722+0000 7fd19e02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T02:37:11.646 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T02:37:11.647 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T02:37:13.647 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T02:37:13.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T02:37:13.761+0000 7f338cb5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T02:39:13.686 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T02:39:13.687 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T02:39:15.687 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T02:39:15.803 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T02:39:15.801+0000 7fadc2b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T02:41:15.725 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T02:41:15.726 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T02:41:17.727 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T02:41:17.843 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T02:41:17.840+0000 7faf35a24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T02:43:17.765 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T02:43:17.766 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T02:43:19.767 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T02:43:19.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T02:43:19.880+0000 7fd67482d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T02:45:19.805 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T02:45:19.806 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T02:45:21.807 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T02:45:21.923 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T02:45:21.921+0000 7fa0ce500640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T02:47:21.845 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T02:47:21.845 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T02:47:23.846 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T02:47:23.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T02:47:23.959+0000 7f6f5159e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T02:49:23.883 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T02:49:23.885 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T02:49:25.886 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T02:49:26.000 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T02:49:25.999+0000 7fa8cbcad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T02:51:25.924 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T02:51:25.925 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T02:51:27.926 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T02:51:28.041 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T02:51:28.040+0000 7f0b7aa64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T02:53:27.963 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T02:53:27.964 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T02:53:29.965 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T02:53:30.080 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T02:53:30.079+0000 7f0cb126d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T02:55:30.003 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T02:55:30.004 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T02:55:32.004 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T02:55:32.120 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T02:55:32.119+0000 7f670c42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T02:57:32.042 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T02:57:32.043 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T02:57:34.044 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T02:57:34.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T02:57:34.157+0000 7f804211e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T02:59:34.081 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T02:59:34.082 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T02:59:36.084 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T02:59:36.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T02:59:36.198+0000 7ff2ed75e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T03:01:36.122 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T03:01:36.123 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T03:01:38.124 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T03:01:38.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T03:01:38.239+0000 7f239379e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T03:03:38.162 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T03:03:38.163 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T03:03:40.164 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T03:03:40.279 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T03:03:40.278+0000 7f262e39e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T03:05:40.202 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T03:05:40.203 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T03:05:42.204 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T03:05:42.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T03:05:42.317+0000 7f5da0a64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T03:07:42.243 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T03:07:42.243 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T03:07:44.245 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T03:07:44.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T03:07:44.358+0000 7f16ded9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T03:09:44.283 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T03:09:44.284 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T03:09:46.285 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T03:09:46.400 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T03:09:46.397+0000 7f163b39e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T03:11:46.322 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T03:11:46.323 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T03:11:48.324 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T03:11:48.440 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T03:11:48.438+0000 7fcf91e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T03:13:48.363 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T03:13:48.364 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T03:13:50.365 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T03:13:50.482 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T03:13:50.479+0000 7f386ae13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T03:15:50.403 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T03:15:50.404 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T03:15:52.405 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T03:15:52.521 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T03:15:52.518+0000 7f7f7526d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T03:17:52.443 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T03:17:52.444 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T03:17:54.446 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T03:17:54.561 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T03:17:54.558+0000 7f3462660640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T03:19:54.483 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T03:19:54.485 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T03:19:56.486 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T03:19:56.602 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T03:19:56.598+0000 7fab1039e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T03:21:56.524 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T03:21:56.525 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T03:21:58.526 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T03:21:58.641 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T03:21:58.637+0000 7f4296264640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T03:23:58.564 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T03:23:58.565 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T03:24:00.566 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T03:24:00.682 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T03:24:00.678+0000 7fb2bdf9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T03:26:00.604 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T03:26:00.605 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T03:26:02.606 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T03:26:02.720 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T03:26:02.715+0000 7ffa6cb1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T03:28:02.643 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T03:28:02.644 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T03:28:04.645 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T03:28:04.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T03:28:04.757+0000 7f992539e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T03:30:04.682 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T03:30:04.683 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T03:30:06.685 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T03:30:06.800 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T03:30:06.796+0000 7f8e03660640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T03:32:06.723 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T03:32:06.724 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T03:32:08.726 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T03:32:08.841 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T03:32:08.838+0000 7fe5a119e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T03:34:08.765 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T03:34:08.766 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T03:34:10.767 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T03:34:10.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T03:34:10.878+0000 7f0087cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T03:36:10.805 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T03:36:10.806 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T03:36:12.807 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T03:36:12.922 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T03:36:12.918+0000 7f39eb95e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T03:38:12.845 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T03:38:12.846 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T03:38:14.847 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T03:38:14.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T03:38:14.958+0000 7f5bcea2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T03:40:14.885 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T03:40:14.886 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T03:40:16.888 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T03:40:17.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T03:40:17.000+0000 7ff60255e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T03:42:16.925 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T03:42:16.926 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T03:42:18.927 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T03:42:19.042 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T03:42:19.039+0000 7f81fbf5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T03:44:18.965 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T03:44:18.966 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T03:44:20.968 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T03:44:21.082 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T03:44:21.080+0000 7f1eb431e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T03:46:21.005 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T03:46:21.006 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T03:46:23.007 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T03:46:23.122 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T03:46:23.120+0000 7f610422d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T03:48:23.045 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T03:48:23.046 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T03:48:25.048 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T03:48:25.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T03:48:25.162+0000 7f06aa515640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T03:50:25.086 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T03:50:25.086 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T03:50:27.088 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T03:50:27.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T03:50:27.202+0000 7feb32915640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T03:52:27.126 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T03:52:27.127 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T03:52:29.129 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T03:52:29.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T03:52:29.243+0000 7f65eb86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T03:54:29.167 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T03:54:29.168 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T03:54:31.169 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T03:54:31.284 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T03:54:31.283+0000 7fb3c079e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T03:56:31.207 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T03:56:31.208 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T03:56:33.210 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T03:56:33.325 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T03:56:33.323+0000 7fbe4c86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T03:58:33.247 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T03:58:33.248 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T03:58:35.250 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T03:58:35.364 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T03:58:35.363+0000 7fcaf32ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T04:00:35.288 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T04:00:35.289 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T04:00:37.291 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T04:00:37.408 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T04:00:37.407+0000 7fad2ae6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T04:02:37.329 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T04:02:37.330 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T04:02:39.331 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T04:02:39.446 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T04:02:39.444+0000 7f266515e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T04:04:39.368 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T04:04:39.369 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T04:04:41.370 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T04:04:41.485 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T04:04:41.484+0000 7f9cbe75e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T04:06:41.409 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T04:06:41.409 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T04:06:43.411 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T04:06:43.528 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T04:06:43.527+0000 7f709d59e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T04:08:43.449 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T04:08:43.450 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T04:08:45.451 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T04:08:45.568 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T04:08:45.566+0000 7efd7162d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T04:10:45.490 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T04:10:45.491 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T04:10:47.492 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T04:10:47.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T04:10:47.605+0000 7feb1dd5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T04:12:47.529 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T04:12:47.555 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T04:12:49.557 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T04:12:49.672 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T04:12:49.671+0000 7f8885b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T04:14:49.595 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T04:14:49.596 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T04:14:51.598 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T04:14:51.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T04:14:51.711+0000 7f3d8a0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T04:16:51.635 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T04:16:51.636 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T04:16:53.637 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T04:16:53.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T04:16:53.753+0000 7fd4d4a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T04:18:53.675 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T04:18:53.676 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T04:18:55.678 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T04:18:55.793 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T04:18:55.792+0000 7f54db500640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T04:20:55.715 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T04:20:55.716 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T04:20:57.717 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T04:20:57.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T04:20:57.832+0000 7f9772b15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T04:22:57.755 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T04:22:57.756 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T04:22:59.757 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T04:22:59.872 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T04:22:59.871+0000 7f19c0f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T04:24:59.795 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T04:24:59.796 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T04:25:01.797 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T04:25:01.912 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T04:25:01.910+0000 7f44d8b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T04:27:01.834 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T04:27:01.835 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T04:27:03.837 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T04:27:03.953 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T04:27:03.952+0000 7fb52ec24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T04:29:03.875 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T04:29:03.876 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T04:29:05.877 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T04:29:05.993 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T04:29:05.991+0000 7f382455e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T04:31:05.914 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T04:31:05.915 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T04:31:07.916 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T04:31:08.030 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T04:31:08.029+0000 7f7530f55640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T04:33:07.953 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T04:33:07.954 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T04:33:09.956 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T04:33:10.071 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T04:33:10.069+0000 7f03a1d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T04:35:09.993 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T04:35:09.994 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T04:35:11.996 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T04:35:12.111 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T04:35:12.109+0000 7f429579e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T04:37:12.033 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T04:37:12.034 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T04:37:14.035 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T04:37:14.152 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T04:37:14.150+0000 7f3f81206640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T04:39:14.073 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T04:39:14.074 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T04:39:16.075 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T04:39:16.191 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T04:39:16.189+0000 7f037b26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T04:41:16.113 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T04:41:16.120 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T04:41:18.121 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T04:41:18.236 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T04:41:18.234+0000 7febb0ade640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T04:43:18.158 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T04:43:18.159 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T04:43:20.160 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T04:43:20.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T04:43:20.274+0000 7fa84395e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T04:45:20.199 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T04:45:20.200 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T04:45:22.201 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T04:45:22.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T04:45:22.309+0000 7f44fa62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T04:47:22.238 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T04:47:22.239 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T04:47:24.240 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T04:47:24.356 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T04:47:24.354+0000 7f6233900640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T04:49:24.278 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T04:49:24.279 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T04:49:26.280 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T04:49:26.396 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T04:49:26.394+0000 7f57fd11e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T04:51:26.318 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T04:51:26.319 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T04:51:28.321 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T04:51:28.437 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T04:51:28.435+0000 7f15b926d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T04:53:28.359 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T04:53:28.360 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T04:53:30.362 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T04:53:30.480 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T04:53:30.478+0000 7f1a98ef0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T04:55:30.400 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T04:55:30.401 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T04:55:32.402 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T04:55:32.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T04:55:32.516+0000 7f98d162d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T04:57:32.440 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T04:57:32.443 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T04:57:34.444 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T04:57:34.559 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T04:57:34.557+0000 7fc71735e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T04:59:34.482 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T04:59:34.483 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T04:59:36.483 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T04:59:36.599 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T04:59:36.597+0000 7f923199e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T05:01:36.522 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T05:01:36.523 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T05:01:38.524 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T05:01:38.640 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T05:01:38.639+0000 7f164255e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T05:03:38.563 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T05:03:38.564 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T05:03:40.565 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T05:03:40.681 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T05:03:40.679+0000 7fda2f0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T05:05:40.603 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T05:05:40.604 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T05:05:42.605 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T05:05:42.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T05:05:42.718+0000 7f75ff15e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T05:07:42.643 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T05:07:42.644 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T05:07:44.645 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T05:07:44.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T05:07:44.759+0000 7f756ee6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T05:09:44.683 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T05:09:44.684 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T05:09:46.686 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T05:09:46.801 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T05:09:46.800+0000 7f1f86cf7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T05:11:46.724 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T05:11:46.725 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T05:11:48.727 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T05:11:48.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T05:11:48.842+0000 7ff2d951e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T05:13:48.765 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T05:13:48.766 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T05:13:50.767 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T05:13:50.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T05:13:50.881+0000 7f99a12ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T05:15:50.805 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T05:15:50.806 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T05:15:52.807 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T05:15:52.923 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T05:15:52.921+0000 7fc30a8f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T05:17:52.845 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T05:17:52.846 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T05:17:54.847 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T05:17:54.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T05:17:54.962+0000 7ff58ea2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T05:19:54.884 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T05:19:54.885 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T05:19:56.887 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T05:19:57.002 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T05:19:57.000+0000 7f773f22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T05:21:56.924 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T05:21:56.925 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T05:21:58.927 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T05:21:59.042 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T05:21:59.040+0000 7f4f27355640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T05:23:58.965 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T05:23:58.968 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T05:24:00.969 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T05:24:01.084 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T05:24:01.081+0000 7eff05017640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T05:26:01.006 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T05:26:01.007 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T05:26:03.008 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T05:26:03.123 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T05:26:03.120+0000 7f9f85c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T05:28:03.046 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T05:28:03.047 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T05:28:05.048 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T05:28:05.163 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T05:28:05.160+0000 7f10e0508640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T05:30:05.086 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T05:30:05.087 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T05:30:07.088 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T05:30:07.203 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T05:30:07.200+0000 7fecbc39e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T05:32:07.126 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T05:32:07.127 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T05:32:09.128 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T05:32:09.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T05:32:09.240+0000 7f63e0c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T05:34:09.167 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T05:34:09.167 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T05:34:11.168 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T05:34:11.283 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T05:34:11.279+0000 7f377cd1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T05:36:11.206 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T05:36:11.207 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T05:36:13.208 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T05:36:13.326 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T05:36:13.323+0000 7f3b5e0de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T05:38:13.247 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T05:38:13.247 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T05:38:15.248 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T05:38:15.365 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T05:38:15.362+0000 7f2d89c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T05:40:15.286 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T05:40:15.287 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T05:40:17.288 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T05:40:17.403 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T05:40:17.400+0000 7f83f2d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T05:42:17.326 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T05:42:17.327 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T05:42:19.328 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T05:42:19.444 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T05:42:19.440+0000 7f5c9cc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T05:44:19.366 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T05:44:19.367 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T05:44:21.368 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T05:44:21.484 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T05:44:21.480+0000 7fa1a7d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T05:46:21.406 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T05:46:21.407 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T05:46:23.408 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T05:46:23.524 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T05:46:23.520+0000 7fad0119e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T05:48:23.445 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T05:48:23.446 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T05:48:25.448 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T05:48:25.564 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T05:48:25.559+0000 7f28e2b1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T05:50:25.485 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T05:50:25.486 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T05:50:27.487 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T05:50:27.603 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T05:50:27.599+0000 7fcf4acde640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T05:52:27.525 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T05:52:27.526 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T05:52:29.527 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T05:52:29.642 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T05:52:29.638+0000 7fe2f866d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T05:54:29.564 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T05:54:29.565 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T05:54:31.566 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T05:54:31.681 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T05:54:31.677+0000 7ffb8079e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T05:56:31.603 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T05:56:31.604 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T05:56:33.605 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T05:56:33.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T05:56:33.718+0000 7faa2c11e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T05:58:33.643 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T05:58:33.644 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T05:58:35.645 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T05:58:35.762 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T05:58:35.759+0000 7f686d75e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T06:00:35.682 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T06:00:35.683 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T06:00:37.684 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T06:00:37.800 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T06:00:37.797+0000 7f091c95e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T06:02:37.721 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T06:02:37.736 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T06:02:39.738 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T06:02:39.854 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T06:02:39.850+0000 7f684c99e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T06:04:39.776 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T06:04:39.777 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T06:04:41.778 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T06:04:41.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T06:04:41.892+0000 7f2ebc19e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T06:06:41.818 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T06:06:41.819 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T06:06:43.820 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T06:06:43.936 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T06:06:43.933+0000 7fe0ad31e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T06:08:43.859 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T06:08:43.860 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T06:08:45.861 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T06:08:45.976 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T06:08:45.973+0000 7f342ce6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T06:10:45.900 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T06:10:45.901 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T06:10:47.902 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T06:10:48.017 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T06:10:48.013+0000 7f10f866d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T06:12:47.940 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T06:12:47.940 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T06:12:49.942 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T06:12:50.059 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T06:12:50.055+0000 7ff0ed35e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T06:14:49.981 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T06:14:50.002 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T06:14:52.003 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T06:14:52.118 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T06:14:52.115+0000 7f9c6eb1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T06:16:52.041 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T06:16:52.042 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T06:16:54.043 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T06:16:54.158 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T06:16:54.154+0000 7f27a1e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T06:18:54.081 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T06:18:54.082 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T06:18:56.083 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T06:18:56.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T06:18:56.195+0000 7f6beb91e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T06:20:56.121 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T06:20:56.122 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T06:20:58.123 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T06:20:58.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T06:20:58.234+0000 7f69f902d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T06:22:58.161 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T06:22:58.162 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T06:23:00.163 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T06:23:00.279 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T06:23:00.275+0000 7fc2e8c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T06:25:00.200 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T06:25:00.201 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T06:25:02.203 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T06:25:02.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T06:25:02.314+0000 7f509a71e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T06:27:02.241 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T06:27:02.242 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T06:27:04.243 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T06:27:04.359 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T06:27:04.355+0000 7f77cd82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T06:29:04.281 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T06:29:04.282 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T06:29:06.284 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T06:29:06.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T06:29:06.396+0000 7f0a3c55e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T06:31:06.320 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T06:31:06.337 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T06:31:08.338 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T06:31:08.454 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T06:31:08.451+0000 7f4e2562d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T06:33:08.375 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T06:33:08.376 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T06:33:10.378 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T06:33:10.494 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T06:33:10.491+0000 7fc0f299e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T06:35:10.415 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T06:35:10.416 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T06:35:12.417 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T06:35:12.532 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T06:35:12.529+0000 7f569742d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T06:37:12.454 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T06:37:12.455 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T06:37:14.456 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T06:37:14.572 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T06:37:14.570+0000 7f2bde99e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T06:39:14.494 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T06:39:14.495 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T06:39:16.496 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T06:39:16.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T06:39:16.609+0000 7fbc6795e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T06:41:16.534 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T06:41:16.535 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T06:41:18.536 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T06:41:18.651 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T06:41:18.649+0000 7f17f915e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T06:43:18.573 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T06:43:18.574 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T06:43:20.575 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T06:43:20.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T06:43:20.692+0000 7f8e1c95e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T06:45:20.616 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T06:45:20.617 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T06:45:22.619 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T06:45:22.734 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T06:45:22.732+0000 7f3d7bd11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T06:47:22.657 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T06:47:22.658 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T06:47:24.660 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T06:47:24.776 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T06:47:24.774+0000 7f5e73f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T06:49:24.698 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T06:49:24.699 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T06:49:26.700 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T06:49:26.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T06:49:26.815+0000 7f8c6eade640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T06:51:26.738 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T06:51:26.739 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T06:51:28.740 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T06:51:28.856 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T06:51:28.854+0000 7fd1c971e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T06:53:28.777 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T06:53:28.778 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T06:53:30.780 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T06:53:30.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T06:53:30.894+0000 7f82c122d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T06:55:30.817 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T06:55:30.818 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T06:55:32.819 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T06:55:32.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T06:55:32.933+0000 7fa52bd5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T06:57:32.857 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T06:57:32.858 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T06:57:34.859 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T06:57:34.975 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T06:57:34.974+0000 7f6c14c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T06:59:34.897 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T06:59:34.899 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T06:59:36.900 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T06:59:37.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T06:59:37.015+0000 7f97cff1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T07:01:36.938 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T07:01:36.939 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T07:01:38.939 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T07:01:39.061 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T07:01:39.059+0000 7f3f7311e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T07:03:38.983 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T07:03:38.984 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T07:03:40.986 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T07:03:41.102 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T07:03:41.101+0000 7fe02ee60640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T07:05:41.024 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T07:05:41.025 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T07:05:43.026 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T07:05:43.142 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T07:05:43.141+0000 7f314051e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T07:07:43.064 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T07:07:43.065 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T07:07:45.066 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T07:07:45.182 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T07:07:45.181+0000 7f773fc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T07:09:45.105 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T07:09:45.111 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T07:09:47.112 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T07:09:47.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T07:09:47.226+0000 7f5c44c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T07:11:47.150 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T07:11:47.151 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T07:11:49.152 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T07:11:49.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T07:11:49.267+0000 7f07b319e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T07:13:49.191 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T07:13:49.192 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T07:13:51.193 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T07:13:51.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T07:13:51.308+0000 7feb45715640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T07:15:51.232 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T07:15:51.233 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T07:15:53.235 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T07:15:53.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T07:15:53.350+0000 7f16e8c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T07:17:53.273 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T07:17:53.273 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T07:17:55.274 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T07:17:55.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T07:17:55.388+0000 7efc29700640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T07:19:55.312 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T07:19:55.313 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T07:19:57.314 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T07:19:57.432 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T07:19:57.430+0000 7f39f76ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T07:21:57.352 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T07:21:57.353 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T07:21:59.354 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T07:21:59.471 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T07:21:59.469+0000 7fab79b1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T07:23:59.394 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T07:24:00.929 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T07:24:02.931 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T07:24:03.046 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T07:24:03.044+0000 7fc201d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T07:26:02.969 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T07:26:02.970 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T07:26:04.971 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T07:26:05.088 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T07:26:05.087+0000 7f8050d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T07:28:05.010 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T07:28:05.011 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T07:28:07.012 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T07:28:07.126 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T07:28:07.125+0000 7f52a5f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T07:30:07.049 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T07:30:07.050 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T07:30:09.052 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T07:30:09.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T07:30:09.166+0000 7f6973b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T07:32:09.090 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T07:32:09.090 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T07:32:11.092 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T07:32:11.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T07:32:11.206+0000 7f7ae7260640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T07:34:11.130 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T07:34:11.131 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T07:34:13.131 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T07:34:13.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T07:34:13.245+0000 7f1850cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T07:36:13.173 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T07:36:13.174 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T07:36:15.175 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T07:36:15.290 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T07:36:15.288+0000 7f97f8f5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T07:38:15.212 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T07:38:15.237 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T07:38:17.238 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T07:38:17.355 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T07:38:17.353+0000 7f3d7b006640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T07:40:17.277 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T07:40:17.278 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T07:40:19.279 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T07:40:19.395 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T07:40:19.394+0000 7fbc9c6de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T07:42:19.317 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T07:42:19.318 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T07:42:21.319 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T07:42:21.435 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T07:42:21.434+0000 7fbdf9d1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T07:44:21.357 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T07:44:21.358 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T07:44:23.359 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T07:44:23.477 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T07:44:23.475+0000 7f2d9d95e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T07:46:23.397 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T07:46:23.398 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T07:46:25.399 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T07:46:25.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T07:46:25.513+0000 7fa5264ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T07:48:25.437 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T07:48:25.437 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T07:48:27.439 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T07:48:27.554 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T07:48:27.553+0000 7f17f7ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T07:50:27.476 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T07:50:27.477 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T07:50:29.478 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T07:50:29.593 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T07:50:29.592+0000 7fcbedd5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T07:52:29.516 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T07:52:29.820 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T07:52:31.821 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T07:52:31.936 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T07:52:31.934+0000 7fb724c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T07:54:31.858 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T07:54:31.859 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T07:54:33.860 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T07:54:33.977 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T07:54:33.975+0000 7f2583e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T07:56:33.901 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T07:56:33.902 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T07:56:35.902 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T07:56:36.019 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T07:56:36.017+0000 7f292331e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T07:58:35.940 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T07:58:35.969 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T07:58:37.970 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T07:58:38.086 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T07:58:38.085+0000 7fc415d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T08:00:38.008 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T08:00:38.009 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T08:00:40.011 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T08:00:40.126 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T08:00:40.125+0000 7fc13115e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T08:02:40.048 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T08:02:40.049 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T08:02:42.050 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T08:02:42.166 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T08:02:42.164+0000 7fedff99e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T08:04:42.088 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T08:04:42.111 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T08:04:44.113 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T08:04:44.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T08:04:44.229+0000 7fe0af111640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T08:06:44.151 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T08:06:44.152 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T08:06:46.154 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T08:06:46.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T08:06:46.268+0000 7f4b99a13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T08:08:46.191 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T08:08:46.192 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T08:08:48.193 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T08:08:48.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T08:08:48.308+0000 7fcb85d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T08:10:48.232 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T08:10:48.232 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T08:10:50.234 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T08:10:50.349 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T08:10:50.348+0000 7f1be8d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T08:12:50.271 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T08:12:50.272 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T08:12:52.273 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T08:12:52.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T08:12:52.388+0000 7f10530ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T08:14:52.311 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T08:14:52.312 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T08:14:54.313 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T08:14:54.428 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T08:14:54.427+0000 7f8813cde640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T08:16:54.351 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T08:16:54.352 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T08:16:56.353 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T08:16:56.474 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T08:16:56.473+0000 7fa0e46ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T08:18:56.396 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T08:18:56.415 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T08:18:58.416 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T08:18:58.531 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T08:18:58.529+0000 7f25a9755640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T08:20:58.454 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T08:20:58.455 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T08:21:00.456 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T08:21:00.571 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T08:21:00.569+0000 7f97bfaad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T08:23:00.495 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T08:23:00.496 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T08:23:02.497 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T08:23:02.612 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T08:23:02.610+0000 7f510b62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T08:25:02.535 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T08:25:02.557 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T08:25:04.558 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T08:25:04.675 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T08:25:04.673+0000 7f2da66ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T08:27:04.596 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T08:27:04.596 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T08:27:06.598 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T08:27:06.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T08:27:06.711+0000 7f295195e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T08:29:06.638 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T08:29:06.639 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T08:29:08.640 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T08:29:08.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T08:29:08.753+0000 7f299d66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T08:31:08.678 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T08:31:08.678 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T08:31:10.680 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T08:31:10.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T08:31:10.793+0000 7fda4d900640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T08:33:10.718 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T08:33:10.718 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T08:33:12.720 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T08:33:12.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T08:33:12.836+0000 7fc6042ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T08:35:12.762 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T08:35:12.763 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T08:35:14.764 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T08:35:14.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T08:35:14.878+0000 7f1267e13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T08:37:14.801 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T08:37:14.802 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T08:37:16.804 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T08:37:16.921 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T08:37:16.919+0000 7f352002d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T08:39:16.844 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T08:39:16.844 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T08:39:18.845 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T08:39:18.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T08:39:18.961+0000 7eff28e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T08:41:18.886 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T08:41:18.888 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T08:41:20.889 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T08:41:21.005 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T08:41:21.003+0000 7fb9b146d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T08:43:20.928 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T08:43:20.929 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T08:43:22.931 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T08:43:23.047 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T08:43:23.046+0000 7fa73111e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T08:45:22.969 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T08:45:22.971 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T08:45:24.972 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T08:45:25.087 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T08:45:25.085+0000 7fc9a199e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T08:47:25.010 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T08:47:25.011 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T08:47:27.013 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T08:47:27.127 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T08:47:27.125+0000 7efe69e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T08:49:27.051 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T08:49:27.051 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T08:49:29.053 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T08:49:29.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T08:49:29.166+0000 7f4b8ce2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T08:51:29.091 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T08:51:29.115 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T08:51:31.116 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T08:51:31.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T08:51:31.230+0000 7fefbf0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T08:53:31.155 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T08:53:31.156 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T08:53:33.157 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T08:53:33.272 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T08:53:33.269+0000 7efcd075e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T08:55:33.195 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T08:55:33.196 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T08:55:35.198 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T08:55:35.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T08:55:35.311+0000 7faf9631e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T08:57:35.236 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T08:57:35.237 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T08:57:37.238 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T08:57:37.354 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T08:57:37.351+0000 7fd70bd9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T08:59:37.276 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T08:59:37.280 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T08:59:39.282 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T08:59:39.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T08:59:39.394+0000 7fcdee59e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T09:01:39.319 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T09:01:39.320 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T09:01:41.321 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T09:01:41.437 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T09:01:41.434+0000 7f07b899e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T09:03:41.360 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T09:03:41.361 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T09:03:43.362 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T09:03:43.478 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T09:03:43.475+0000 7f322d4f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T09:05:43.401 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T09:05:43.403 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T09:05:45.404 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T09:05:45.519 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T09:05:45.516+0000 7fada179e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T09:07:45.441 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T09:07:45.442 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T09:07:47.443 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T09:07:47.559 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T09:07:47.556+0000 7f707f79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T09:09:47.481 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T09:09:47.483 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T09:09:49.484 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T09:09:49.599 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T09:09:49.595+0000 7fe68039e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T09:11:49.522 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T09:11:49.523 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T09:11:51.525 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T09:11:51.641 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T09:11:51.638+0000 7f969a79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T09:13:51.563 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T09:13:51.564 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T09:13:53.565 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T09:13:53.681 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T09:13:53.678+0000 7fbd79f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T09:15:53.603 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T09:15:53.604 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T09:15:55.606 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T09:15:55.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T09:15:55.720+0000 7f6aa94ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T09:17:55.643 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T09:17:55.644 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T09:17:57.645 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T09:17:57.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T09:17:57.760+0000 7f898419e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T09:19:57.684 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T09:19:57.685 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T09:19:59.686 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T09:19:59.802 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T09:19:59.799+0000 7f17bef9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T09:21:59.724 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T09:21:59.725 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T09:22:01.726 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T09:22:01.843 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T09:22:01.841+0000 7f9bfdc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T09:24:01.764 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T09:24:01.765 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T09:24:03.766 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T09:24:03.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T09:24:03.880+0000 7efdaf355640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T09:26:03.804 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T09:26:03.805 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T09:26:05.806 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T09:26:05.922 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T09:26:05.920+0000 7f5726d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T09:28:05.844 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T09:28:05.845 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T09:28:07.846 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T09:28:07.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T09:28:07.960+0000 7f4f7562d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T09:30:07.884 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T09:30:07.885 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T09:30:09.886 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T09:30:10.001 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T09:30:09.999+0000 7efe94515640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T09:32:09.923 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T09:32:09.924 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T09:32:11.926 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T09:32:12.042 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T09:32:12.040+0000 7fbdc4ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T09:34:11.964 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T09:34:11.965 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T09:34:13.965 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T09:34:14.082 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T09:34:14.080+0000 7fba41d00640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T09:36:14.004 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T09:36:14.005 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T09:36:16.006 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T09:36:16.121 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T09:36:16.120+0000 7fddcf46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T09:38:16.044 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T09:38:16.045 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T09:38:18.046 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T09:38:18.162 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T09:38:18.160+0000 7f924159e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T09:40:18.084 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T09:40:18.085 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T09:40:20.087 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T09:40:20.202 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T09:40:20.201+0000 7fcf68e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T09:42:20.125 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T09:42:20.126 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T09:42:22.127 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T09:42:22.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T09:42:22.242+0000 7fd7b4a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T09:44:22.165 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T09:44:22.166 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T09:44:24.167 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T09:44:24.283 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T09:44:24.282+0000 7f006ad5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T09:46:24.205 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T09:46:24.206 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T09:46:26.207 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T09:46:26.322 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T09:46:26.321+0000 7f29b0b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T09:48:26.245 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T09:48:26.246 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T09:48:28.247 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T09:48:28.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T09:48:28.360+0000 7f47a8460640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T09:50:28.284 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T09:50:28.285 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T09:50:30.285 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T09:50:30.402 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T09:50:30.400+0000 7fb94fcad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T09:52:30.323 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T09:52:30.324 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T09:52:32.326 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T09:52:32.442 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T09:52:32.440+0000 7f979599e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T09:54:32.363 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T09:54:32.364 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T09:54:34.366 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T09:54:34.481 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T09:54:34.481+0000 7f3466d1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T09:56:34.402 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T09:56:34.403 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T09:56:36.405 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T09:56:36.520 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T09:56:36.520+0000 7f4a24915640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T09:58:36.442 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T09:58:36.443 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T09:58:38.444 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T09:58:38.560 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T09:58:38.561+0000 7f3c2382d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T10:00:38.482 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T10:00:38.483 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T10:00:40.485 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T10:00:40.601 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T10:00:40.602+0000 7f9e89ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T10:02:40.523 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T10:02:40.524 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T10:02:42.525 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T10:02:42.641 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T10:02:42.642+0000 7f7bc155e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T10:04:42.563 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T10:04:42.564 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T10:04:44.565 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T10:04:44.681 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T10:04:44.683+0000 7fcfac91e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T10:06:44.603 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T10:06:44.604 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T10:06:46.605 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T10:06:46.720 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T10:06:46.723+0000 7fb18922d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T10:08:46.643 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T10:08:46.644 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T10:08:48.645 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T10:08:48.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T10:08:48.763+0000 7f6a08b1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T10:10:48.683 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T10:10:48.684 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T10:10:50.685 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T10:10:50.801 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T10:10:50.804+0000 7fb14f2de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T10:12:50.722 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T10:12:50.723 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T10:12:52.725 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T10:12:52.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T10:12:52.843+0000 7fc3c6c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T10:14:52.763 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T10:14:52.764 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T10:14:54.765 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T10:14:54.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T10:14:54.884+0000 7f2dacf1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T10:16:54.804 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T10:16:54.805 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T10:16:56.806 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T10:16:56.921 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T10:16:56.925+0000 7f54ed39e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T10:18:56.843 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T10:18:56.844 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T10:18:58.845 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T10:18:58.960 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T10:18:58.964+0000 7f416ff9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T10:20:58.882 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T10:20:58.883 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T10:21:00.885 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T10:21:01.002 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T10:21:01.006+0000 7fa1dea2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T10:23:00.922 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T10:23:00.923 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T10:23:02.925 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T10:23:03.040 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T10:23:03.044+0000 7f811c62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T10:25:02.963 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T10:25:02.964 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T10:25:04.965 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T10:25:05.080 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T10:25:05.085+0000 7ff88451e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T10:27:05.002 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T10:27:05.003 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T10:27:07.004 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T10:27:07.119 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T10:27:07.123+0000 7fb49ef9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T10:29:07.042 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T10:29:07.043 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T10:29:09.043 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T10:29:09.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T10:29:09.162+0000 7f8be191e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T10:31:09.081 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T10:31:09.082 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T10:31:11.083 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T10:31:11.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T10:31:11.203+0000 7f6474515640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T10:33:11.121 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T10:33:11.122 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T10:33:13.123 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T10:33:13.239 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T10:33:13.244+0000 7fc3982ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T10:35:13.161 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T10:35:13.162 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T10:35:15.163 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T10:35:15.280 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T10:35:15.285+0000 7f747c79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T10:37:15.201 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T10:37:15.202 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T10:37:17.203 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T10:37:17.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T10:37:17.324+0000 7fd15855e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T10:39:17.240 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T10:39:17.241 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T10:39:19.243 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-12-16T10:39:19.360 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T10:39:19.366+0000 7fd3c086d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T10:41:19.280 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T10:41:19.281 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-12-16T10:41:21.282 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-12-16T10:41:21.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-12-16T10:41:21.404+0000 7f9ed5aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-16T10:41:42.757 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2023-12-16T10:41:42.758 DEBUG:teuthology.task.console_log:Killing console logger for smithi071 2023-12-16T10:41:42.758 DEBUG:teuthology.task.console_log:Killing console logger for smithi072 2023-12-16T10:41:42.758 DEBUG:teuthology.exit:Finished running handlers